Untitled
Alignment Lab AI Lab Assistant model draft
unknown
plain_text
2 years ago
8.9 kB
25
Indexable
http://arxiv.org/abs/2303.06689 https://arxiv.org/abs/2203.11171 https://arxiv.org/abs/2303.17071 http://arxiv.org/abs/2303.17071v1 https://arxiv.org/abs/2211.01910 https://arxiv.org/abs/2305.02897 https://arxiv.org/pdf/2108.08877.pdf https://arxiv.org/pdf/2006.08671.pdf https://arxiv.org/pdf/2009.09634.pdf https://arxiv.org/abs/2304.11477 http://arxiv.org/abs/2303.09014 http://arxiv.org/abs/2303.17491 http://arxiv.org/abs/2304.05128 http://arxiv.org/abs/2304.08103 https://arxiv.org/ftp/arxiv/papers/2303/2303.17482.pdf https://paperswithcode.com/dataset/dr-bench https://arxiv.org/abs/2205.11916 http://arxiv.org/abs/1901.02860 https://arxiv.org/abs/2207.06881 https://paperswithcode.com/method/absolute-position-encodings https://arxiv.org/abs/2205.05131v1 https://arxiv.org/abs/2305.01625 https://arxiv.org/pdf/2304.15004.pdf https://arxiv.org/abs/2302.07842 https://openreview.net/forum?id=1ikK0kHjvj https://openreview.net/pdf?id=ByME42AqK7 https://arxiv.org/abs/2104.09864 https://arxiv.org/abs/2112.09118 https://arxiv.org/abs/2305.06161 https://arxiv.org/abs/2305.08291 https://log10.io/ https://neuralmagic.com/deepsparse/ https://gpt-index.readthedocs.io/ https://python.langchain.com/ https://crfm.stanford.edu/helm/latest https://instruction-tuning-with-gpt-4.github.io/ https://github.com/project-baize/baize-chatbot https://github.com/curai/curai-research/tree/main/DERA https://github.com/bhargaviparanjape/language-programmes/ https://github.com/ofirpress/attention_with_linear_biases https://github.com/lucidrains https://github.com/lm-sys/ https://github.com/Alignment-Lab-AI/FOP https://github.com/Alignment-Lab-AI/DRCLib https://github.com/Alignment-Lab-AI/TEAL https://github.com/Alignment-Lab-AI/dino https://github.com/Alignment-Lab-AI/mt-dnn https://github.com/Alignment-Lab-AI/identifiable-transformers https://github.com/Alignment-Lab-AI/FLAN https://github.com/Alignment-Lab-AI/LMOps https://github.com/Lightning-AI/lightning https://github.com/booydar/LM-RMT https://github.com/lucidrains/recurrent-memory-transformer-pytorch https://github.com/Cranial-XIX/llm-pddl https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-chat https://github.com/microsoft/deepspeed-mii https://github.com/hpcaitech/ColossalAI https://github.com/MouseAndKeyboard/SmartGPT https://github.com/Alignment-Lab-AI/task_vectors https://github.com/smol-ai/developer https://github.com/IBM/Dromedary/tree/main https://github.com/microsoft/TaskMatrix/tree/main/LowCodeLLM https://github.com/microsoft/guidance https://github.com/posgnu/rci-agent https://github.com/lucidrains/recurrent-memory-transformer-pytorch https://github.com/Cranial-XIX/llm-pddl https://github.com/bhargaviparanjape/language-programmes https://github.com/curai/curai-research/tree/main/DERA https://github.com/chakkaradeep/pyCodeAGI/ https://github.com/mosaicml/streaming https://github.com/AkariAsai/learning_to_retrieve_reasoning_paths https://github.com/bigcode-project/Megatron-LM https://github.com/CarperAI/trlx/tree/main/examples/randomwalks https://huggingface.co/datasets/codeparrot/github-code-clean https://huggingface.co/facebook/galactica-120b https://huggingface.co/Udoy/tomekkorbak-python-github-code-tokenizer https://huggingface.co/groov/code-search-net-tokenizer https://huggingface.co/tczhang/sample-python-code-tokenizer https://huggingface.co/datasets/facebook/content_rephrasing https://huggingface.co/datasets/universal_morphologies https://huggingface.co/models?dataset=dataset:empathetic_dialogues https://huggingface.co/CarperAI/FIM-NeoX-1.3B https://huggingface.co/dorkai/codeX-1.0 https://huggingface.co/datasets/baizhi002/python3.10.8 https://huggingface.co/datasets/datablations/python-megatron https://huggingface.co/datasets/Nan-Do/instructional_code-search-net-python https://huggingface.co/datasets/trelent/the-stack-dedup-python-docstrings-1.0-percent-unified https://huggingface.co/datasets/reshinthadith/synthetic_program_synthesis_python_1M https://huggingface.co/datasets/loubnabnl/python_comment_code_ratio_08 https://huggingface.co/datasets/Dahoas/code-review-instruct-critique-revision-python https://huggingface.co/datasets/Nan-Do/code-search-net-python https://huggingface.co/datasets/Sridevi/python_textbooks https://huggingface.co/datasets/sia-precision-education/pile_python https://huggingface.co/datasets/notional/notional-python https://huggingface.co/datasets/semeru/code-text-python https://huggingface.co/datasets/formermagic/github_python_1m https://huggingface.co/datasets/Fraser/python-state-changes https://huggingface.co/datasets/bigcode/the-stack-dedup https://huggingface.co/datasets/EleutherAI/arithmetic https://huggingface.co/datasets/stanfordnlp/SHP https://huggingface.co/datasets/lighteval/synthetic_reasoning_natural https://huggingface.co/datasets/lintang/numerical_reasoning_arithmetic https://huggingface.co/datasets/lighteval/synthetic_reasoning/ https://huggingface.co/datasets/jxu124/llava_complex_reasoning_77k/ https://huggingface.co/wordcab/llama-natural-instructions-13b https://huggingface.co/datasets/Alignment-Lab-AI/AILabAssistant/blob/main/pretraining%20sets https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered https://huggingface.co/datasets/tasksource/mmlu https://huggingface.co/datasets/QingyiSi/Alpaca-CoT/tree/main/GPTeacher https://huggingface.co/datasets/QingyiSi/Alpaca-CoT/tree/main/dolly https://huggingface.co/datasets/QingyiSi/Alpaca-CoT/tree/main/instruct https://huggingface.co/datasets/QingyiSi/Alpaca-CoT/tree/main/MOSS https://huggingface.co/datasets/QingyiSi/Alpaca-CoT/tree/main/CodeAlpaca https://huggingface.co/datasets/QingyiSi/Alpaca-CoT/tree/main/Chinese-instruction-collection more avaliable upon request, sorry for the disorder i wasnt expecting to suddenly have a reason to compile it all in a readable format, there is more the project draws from but many of these contribute very little to the overall structure and this is additionally missing the pretraining data links and much of my own polished dataset made for the purpose due to local storage. https://www.canva.com/design/DAFjnCZymIo/xH_bnShuEVwZIKVZRIwRtw/edit?utm_content=DAFjnCZymIo&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton this is a hasty reorganization of the documentation i have been accumulating and studying for the last year or so, thr project has been thuroughly discussed with people of various level of skill and ranking within the industry, many of whom will be avaliable to help fill gaps in my skill as neccesary. ill post a more detailed version on the canva but the rough workflow involes a pretraining set of books structured to teach the logical releationship between human expression, logic, and math within the domain of language, structure based loosely off of the related logic in https://en.wikipedia.org/wiki/Laws_of_Form , (excessively tokenized with performant language and python models, optimized embedding relationships represented initially planned when time was not a factor for model explainability)a blended dataset with weighted ratios of python staging up from general understanding to high accuracy at programming instructions that demonstrate the recursive code skeleton steps (a method for developing code bases in a size agnostic way demonstrated in canva) while mixing in a portion of data through out containing moral heuristics, rlhf, natural instruct, chain of thought, tool use, synthetic logic, abstractive logic, instruction following, kindness, helpfulness, and a bunch of SCIENCE, with seqlength amplitude modulation to hopefully optimize performance on the chosen attention mechanisms if any are used. i will likely realize i forgot several things after sending this, updates or link to updates will be posted at the canva link until a github is compiled when time allows, ive also been up for a long time so if you see this tonight and its not updated please be patient ive been preparing for the anthropic hackathon. :) i was making the canva as i saw the link posted fortunately enough! it contains/will contain a flow chart of the initial draft for the training process, scripts used to collect the unique programming training data required, an example of the code handling structure and additional data that is not present in this partial list. including pretraining structure, solution to coding, solution to instruct decoherency, possible solution to hallucination, and new optimization techniques some of which i only fleshed out a few days ago, and predicted results of training. several potential methods for rendering context limit arbitrary with linear scaling to input length. more documentation is avaliable upon request, including current funding sources, the other open sourced developers involved both directly and tangentially - happy to answer any questions as they come up :)