When I was still an academic, I used to conduct several behavioral experiments as part of my research. In these studies, I and plenty of other researchers would always spend a long time designing how the experiment would be conducted, from recruitment to the materials to the analysis. And in this process, the aim was always to conduct a clean experiment, where the effects that one might observe is not affected by factors that we don't care about, also known as confounds. Researchers also tend to be strict when it comes to recruitment, only enrolling participants that definitely fit their enrollment criteria (male, right-handed, 18-64 years of age, no history of visual impediment, for example). However, outside of the academic context, there are other factors that come to play so when one wants to conduct an experiment, then there are other issues one needs to take care of.
I just finished conducting a series of AB testing in my workplace. I won't tell what I was testing here, but nevertheless, I can say that it was a different type of "lab" so to speak. For example, recruitment was an issue. Back when I was an academic, we pretty much thought that there is an endless supply of college students who are willing to be participants in our experiments as long as we pay them a small sum afterwards. Now that is not the case, and so we need to budget our studies given the short supply of participants.
That is just one example, but for the most part, I find myself having two standards, an ideal one, recommended by science; and a pragmatic one, limited by the resources one has (or does not have). And in my current workplace, academics are reaching out to us in some contexts, but I do see conflicting goals every now and then. Primarily, it is because what is important in academia is to find out what wasn't known before, while sometimes what is important for a company is to simply do something that works, whether that is already known for a while or not.
Anyway, I must say that the change of mindset coming out of academia is engaging me. It's like doing what I have been doing before, but with a different set of constraints this time. And I must say, I am having fun finding out what constraints work the best given the constantly changing circumstances.
