Fun stuff. I have no clue if there are attributes to each point in the data set your working with, but if not and if colorizing it is a next step your considering then maybe do it by calculating the centroid and use vector(I,j,k)from that to select a 16-bit rgb color?
That's a good thought! No there are no other attributes in this particular dataset but I can use more of the data from flywire if I feel like doing so, but the more dimensions we're dealing with, the RAM usage is gonna go up like crazy haha. So I might try your advice.
Finding the limits of software and hardware is fun. I ran into CPU limits on some fun tests I was doing for coupled groundwater, surface water, fate and transport interactions with MODFLOW6 and ModelMuse (USGS free software). It’s amazing what you can do with it with a LiDar data set snd some GIS lines and polygons.
It worked ok for big cells dimensions, but I wanted to model entire watersheds at a reasonable scale. When I looked at why it was running so slow I noticed it was using 8% of the CPU and 100% RAM utilization plus a big page swap. Then I realized the core issue is that the software was not written in a language capable of multi threading (using more than one processor, mine had 12). So then I get frustrated, and what do you know the next big thing on USGS’s list is to rewrite that software to take advantage of multi core processors. Of course I could have bought SAS to shortcut the wait, but I was doing this for fun and I found the limits and that was the whole point of the exercise.
Then I found whiteboxtools python code and said wholly 💩 they finally did what I was trying to do 25 years ago. And it’s multithreaded. Why learn how to code Rust if someone already created a python api to harness it?
Oof, I cut my teeth on representations of graph networks much, much smaller than this (thousands of connections) many years ago, and even then it was clear how critical node affinity/placement was. Good job!
"Well, to be more specific, I have a map of every connection between one neuron and another in the brain of a fruit fly, including the strength of those connections." Really? Do neuroscientists know how all the neurons are connected to each other and what the strength of those connections is? If so, my mind is blown, I had no idea that scientists were capable of understanding brains at that level of detail. How do they figure that out?
It's actually a really cool thing. So for a long time, scientists have been excavating layers of brains of various creatures using electron microscopy and staining techniques. Essentially, they go layer by layer, slicing the tiniest slices and then running an electron microscope over each slice. They've then had people as well as automated tools crawling all over that massive data set and mapping every neuron and its pathways and synaptic connections. This dataset is basically the result of that process. They're still going. mapping the types of neuron (they're crowd-sourcing that stuff too, you can sign up and join in if you like!)
SO.... does the blue fuzzy thing resemble the morphology of the actual brain? How did you map all the neurons and connections in the first place - I mean - how do you know what nuerone connects to which other one?
it could be handy if you can make a functional neural network - you could like... I dunno, build an app on your phone or something that could tell you if avocados were perfectly ripe without squeezing them?
I am already thinking of the HORROR BASED APPLICATIONS for this....
but seriously - what is the goal of this work, IRL? genuinely interested. When i studied zoology at uni back in 1988.... one of my lecturers was modelling fly brains from electron microscopy - for some reason he was building relatively huge models of them out of layer upon layer of pink wax which he displayed in a box - was never sure of the point of this, but he was very excited about it.
So that stuff your lecturer was doing, that's how we know which neuron is connected to which other one :) I'm not doing morphology, no, this is purely just modelling the connections. For me, I'm just having fun. The data comes from the Flywire team, who are doing the real crazy stuff: https://flywire.ai
Wait, there are smarty pants that can be read about? :D
Gosh it is super cool though. I've been thinking about ways I might be able to turn it into a functioning neural network, but that's my super-duper-extremely-ambitious part which is unlikely to actually occur.
I think that is fascinating! Also, I just read your piece about conspiratorial thinking and we share some similarities. I watched loose change so many times and got sprung off down some rabbit holes about all the bilderberg stuff and secret societies. I wasn't like a zealot, but I did lend a lot more credence than I would have if I, again, hadn't been raised in an evangelical family. Also, got a short time I was in the vaccine hesitant camp. Maybe in the early aughts for a few years. Only because there were so many people talking about it and I was a new mom and the idea of doing something possibly harmful to my child was terrifying. Thankfully I never graduated to full blown anti vaxxer and I came around way before covid happened.
It's such a big thing in evangelical communities somehow. I remember watching a video of some guy claiming he used to be a satanist who would conjure up the devil and have discussions with him, cast spells, and do all sorts of weird sex rituals, but of course eventually converted and whatnot. I watched so much seventh day adventist nonsense too. Everything in the world of evangelical christianity seemed geared heavily towards the idea of conspiracies happening all around us, all the time, always aimed at "gods chosen people". There was one point where I was convinced anyone who wore a certain amount of black clothing must be a satanist lol.
What a strange world this is :D thank you for sharing your experiences too!
Of course! It's always nice when you read something from another person that mirrors your experiences even if those experiences weren't the most pleasant. I look forward to reading more of what you've e written. 😊
Okay, that is cool!!
Thank you! I thought so too!! Let's see how far we can take this :)
Fun stuff. I have no clue if there are attributes to each point in the data set your working with, but if not and if colorizing it is a next step your considering then maybe do it by calculating the centroid and use vector(I,j,k)from that to select a 16-bit rgb color?
That's a good thought! No there are no other attributes in this particular dataset but I can use more of the data from flywire if I feel like doing so, but the more dimensions we're dealing with, the RAM usage is gonna go up like crazy haha. So I might try your advice.
Finding the limits of software and hardware is fun. I ran into CPU limits on some fun tests I was doing for coupled groundwater, surface water, fate and transport interactions with MODFLOW6 and ModelMuse (USGS free software). It’s amazing what you can do with it with a LiDar data set snd some GIS lines and polygons.
It worked ok for big cells dimensions, but I wanted to model entire watersheds at a reasonable scale. When I looked at why it was running so slow I noticed it was using 8% of the CPU and 100% RAM utilization plus a big page swap. Then I realized the core issue is that the software was not written in a language capable of multi threading (using more than one processor, mine had 12). So then I get frustrated, and what do you know the next big thing on USGS’s list is to rewrite that software to take advantage of multi core processors. Of course I could have bought SAS to shortcut the wait, but I was doing this for fun and I found the limits and that was the whole point of the exercise.
Then I found whiteboxtools python code and said wholly 💩 they finally did what I was trying to do 25 years ago. And it’s multithreaded. Why learn how to code Rust if someone already created a python api to harness it?
It’s about the journey not the results.
Oof, I cut my teeth on representations of graph networks much, much smaller than this (thousands of connections) many years ago, and even then it was clear how critical node affinity/placement was. Good job!
Thats awesome man. I'd love to see the repo when you make it.
"Well, to be more specific, I have a map of every connection between one neuron and another in the brain of a fruit fly, including the strength of those connections." Really? Do neuroscientists know how all the neurons are connected to each other and what the strength of those connections is? If so, my mind is blown, I had no idea that scientists were capable of understanding brains at that level of detail. How do they figure that out?
It's actually a really cool thing. So for a long time, scientists have been excavating layers of brains of various creatures using electron microscopy and staining techniques. Essentially, they go layer by layer, slicing the tiniest slices and then running an electron microscope over each slice. They've then had people as well as automated tools crawling all over that massive data set and mapping every neuron and its pathways and synaptic connections. This dataset is basically the result of that process. They're still going. mapping the types of neuron (they're crowd-sourcing that stuff too, you can sign up and join in if you like!)
This is just crazy cool science :D
SO.... does the blue fuzzy thing resemble the morphology of the actual brain? How did you map all the neurons and connections in the first place - I mean - how do you know what nuerone connects to which other one?
it could be handy if you can make a functional neural network - you could like... I dunno, build an app on your phone or something that could tell you if avocados were perfectly ripe without squeezing them?
I am already thinking of the HORROR BASED APPLICATIONS for this....
but seriously - what is the goal of this work, IRL? genuinely interested. When i studied zoology at uni back in 1988.... one of my lecturers was modelling fly brains from electron microscopy - for some reason he was building relatively huge models of them out of layer upon layer of pink wax which he displayed in a box - was never sure of the point of this, but he was very excited about it.
So that stuff your lecturer was doing, that's how we know which neuron is connected to which other one :) I'm not doing morphology, no, this is purely just modelling the connections. For me, I'm just having fun. The data comes from the Flywire team, who are doing the real crazy stuff: https://flywire.ai
You're like one of those smarty pants that I've been reading about aren't you? Lol. Seriously, that is so freakin cool.
Wait, there are smarty pants that can be read about? :D
Gosh it is super cool though. I've been thinking about ways I might be able to turn it into a functioning neural network, but that's my super-duper-extremely-ambitious part which is unlikely to actually occur.
I think that is fascinating! Also, I just read your piece about conspiratorial thinking and we share some similarities. I watched loose change so many times and got sprung off down some rabbit holes about all the bilderberg stuff and secret societies. I wasn't like a zealot, but I did lend a lot more credence than I would have if I, again, hadn't been raised in an evangelical family. Also, got a short time I was in the vaccine hesitant camp. Maybe in the early aughts for a few years. Only because there were so many people talking about it and I was a new mom and the idea of doing something possibly harmful to my child was terrifying. Thankfully I never graduated to full blown anti vaxxer and I came around way before covid happened.
It's such a big thing in evangelical communities somehow. I remember watching a video of some guy claiming he used to be a satanist who would conjure up the devil and have discussions with him, cast spells, and do all sorts of weird sex rituals, but of course eventually converted and whatnot. I watched so much seventh day adventist nonsense too. Everything in the world of evangelical christianity seemed geared heavily towards the idea of conspiracies happening all around us, all the time, always aimed at "gods chosen people". There was one point where I was convinced anyone who wore a certain amount of black clothing must be a satanist lol.
What a strange world this is :D thank you for sharing your experiences too!
Of course! It's always nice when you read something from another person that mirrors your experiences even if those experiences weren't the most pleasant. I look forward to reading more of what you've e written. 😊