topic_modelling / funcs /clean_funcs.py

Commit History

Rearranged functions for embeddings creation to be compatible with zero GPU space. Updated packages.
cc495e1

seanpedrickcase commited on

Improved initial clean options. Now has option to return embeddings only.
89c4d20

seanpedrickcase commited on

Only aggregate topics not 'other', allowed for minimum sentence length, default max_topics now will auto aggregate topics. Added Cognito Auth functionality (boto3 with AWS).
1e2bb3e

seanpedrickcase commited on

Can split passages into sentences. Improved embedding, LLM representation models, improved zero shot capabilities
55f0ce3

seanpedrickcase commited on

Minor cleaning, csv formatting changes
d80c8f5

Sean-Case commited on

Reduce outliers now more efficient and relabels with correct vectoriser. Default topic labels now tidier. Hiearchical topics outputs more useful for joining to df afterwards. Switched low resource reduction algorithm to UMAP as default is not good.
e1c1f68

Sonnyjim commited on

Should now parse custom regex correctly. Will now wipe previously created embeddings if 'low resource mode' option switched.
0a543a0

Sean-Case commited on

Allowed for uploading custom regex for cleaning. Fixed calculate all probabilities, reduce outliers. Added text tree for hierarchical modelling.
381f959

Sonnyjim commited on

Added clean data options, improved re-representation options and visualisation. General format changes
4effac0

Sonnyjim commited on