added sampling_method to StreamingTextDataset

[snarayan21]

Previously, StreamingTextDataset didn't have the sampling_method arg. Now it does 🤯🤯

[dakinggg]

while you're here...could you pipe through here also?

llm-foundry/llmfoundry/data/finetuning/dataloader.py

Lines 119 to 133 in 68448b2

	dataset = dataset_constructor.build_from_streaming(
	tokenizer=tokenizer,
	local=cfg.dataset.local,
	remote=cfg.dataset.get('remote', None),
	split=cfg.dataset.get('split'),
	shuffle=cfg.dataset.get('shuffle', False),
	predownload=cfg.dataset.get('predownload', 100_000),
	keep_zip=cfg.dataset.get('keep_zip', False),
	download_retry=cfg.dataset.get('download_retry', 2),
	download_timeout=cfg.dataset.get('download_timeout', 60),
	validate_hash=cfg.dataset.get('validate_hash', None),
	shuffle_seed=cfg.dataset.get('shuffle_seed', 9176),
	num_canonical_nodes=cfg.dataset.get('num_canonical_nodes', 128),
	batch_size=device_batch_size,
	)

. I think there actually a few streaming args missing along that code path.

[dakinggg]

LGTM