Hello World!
Hello, and welcome to my blog! I will be documenting my journey in exploring AI and machine learning.
Who am I?
My name is Anas Almasri, an undergraduate student at the University of Toronto. I am specializing in Computer Science, Mathematical Sciences and minoring in Statistics. Currently I am researching at the Toronto Intelligent Systems Laboratory led by Igor Gilitschenski. Outside of my research I also work in Software Engineering. I recently completed an internship at Bitarco during the summer, and I will be joining Amicare this winter.
New Research Interest
Ever since LLMs came out I was always fascinated by how a bunch of numbers in a matrix can create such coherent text. Recently I watched a video about the AlexNet paper by Welch Labs titled "The moment we stopped understanding AI [AlexNet]" (linked below). In the video a glimpse of what the numbers in the matrices represent is visualized by showing the activation of convolution kernels in real-time. For example you were able to tell that a specific convolution kernel activates when it sees a face, which implies that this part of the matrix handles face detection. I want to explore if it is possible to transfer this method of interpretability from image classification models to LLMs. If we can successfully decode what these values represent, the potential goes far beyond just observation. By analyzing the numbers, we could potentially make LLM training far more efficient, or perhaps, even bypass traditional training entirely and build models using only these calculated weights, no data required!
My Future Plans
My next step is to train or fine tune my own small LLM and play around with the parameters to see if I can get any hints on what each number/block represents. I also plan to experiment with 'working backwards' from the result. Instead of just watching the input turn into output, I want to try tracing a specific generated word back through the hidden layers to pinpoint which neurons fired to create it.
Another idea I had is to analyze the actual vectors of closely related tokens, like 'House' and 'Mansion'. I want to see if subtracting these vectors reveals a relationship. For instance, maybe the mathematical difference between them isolates a specific value that represents 'size' within the matrix.
If you have any ideas please feel free to leave a comment below or send me an email. Hope you enjoyed reading and stay tuned for my next post!