KITFORMA · EDITORIAL ANSWER
How can I read a large text file without loading the whole file into memory?
KitForma editorial guide. Process records incrementally without building a full list, and bound unusually long lines before parsing them.
Step-by-step answer
import json
import tempfile
from pathlib import Path
def total_amount(path):
count = total = 0
with path.open(encoding="utf-8") as handle:
while line := handle.readline(1025):
if len(line) > 1024:
raise ValueError("Record exceeds 1024 characters")
total += json.loads(line)["amount"]
count += 1
return count, total
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "amounts.ndjson"
path.write_text('{"amount":2}\n{"amount":3}\n', encoding="utf-8")
print(*total_amount(path))
path.write_text("x" * 1025, encoding="utf-8")
try:
total_amount(path)
except ValueError:
print("oversized record rejected")2 5
oversized record rejectedSources and verification
Sources checked:
This editorial answer was prepared by KitForma with AI assistance. It is not presented as a real member question or an independent user review. Check the sources and the result with your own file; report corrections in the discussion.