2 Why so limiting?

LLMs are text prediction models - nothing more, nothing less. They are built by ingesting large quantities of human-generated text-based data and using some rather sophisticated matrix algebra, non-linear transformations and calculus (truly, these models are a mathematical accomplishment). But, at the the end of the day, they produce content based on the most likely string of words for a given query (mathematicians, I know I’m oversimplifying things!! Don’t come at me!!!). They are built on human knowledge, human errors and human biases and hence can amplify all of those joys and sorrows of humanity. And that’s it. That’s all they do. In some ways, an LMM is combination glorified search engine/turbocharged auto-complete, except it needs the power of an entire nuclear reactor to do its job.

I know some scientists ask ChatGPT how to statistically analyze experiments. This is not a terrible starting point for statistical analysis, but given hallucinations, I hope those scientists are checking those results. One scientist boasted to me that he could upload data straight into chatGPT and it would analyze it for him. I’m not sure where to start with what is wrong with that process but most certainly, scientists need to know and supervise the statistical model used to analyze their results! This is science 101, my peeps.

As mentioned, LLMs are vastly over-hyped. Sam Altman infamously stated at one point that AI would run ‘experiments’ for us scientists while we slept, enabling leaps and bounds in scientific discovery (I cannot find the original tweet, one that was heavily lambasted by the academic research community but here is a statement from former Google CEO Eric Schmidt making the same claim). This idea seems dubious to me, mostly because generative AI cannot truly think; its advantage lies in aggregating information.

4 The Statistician’s View

A few months ago, I attended the Joint Statistical Meetings (the “JSM” by those of us in the know) put on by the American Statistical Association in Portland, Oregon. The biggest topic of discussion among JSM attendees was AI (largely generative AI) and their underlying architecture, large language models. Are LLMs replacing us as statisticians? Are we all about to become obsolete and should we pivot our careers to production engineer (an actual suggestion I heard at the meetings)?

There was a range of views on the matter: AI researchers in particular seemed to view their craft as the inevitable dominant path for the foreseeable future. A keynote delivered by Jason Matheny of the Rand Corporation rattled off a long list of ways statisticians can and should contribute to the AI landscape. Other attendees remain highly skeptical. One staunch critic, Victor Solo of the University of New South Wales, called LLMs “over hyped” and “pre-scientific”. He gave a talk to a packed, standing room-only audience, objecting to the brute force methodology and over-reliance on empiricism of LLMs over domain knowledge and scientific principles. Many others were more circumspect: LLMs are here whether we like them or not, so how do we adapt? (This is largely my view, but with a healthy dose of skepticism added).

There were innumerable talks about using deep learning (another term for AI), but not specifically generative AI to improve the quality of statistical estimation. I don’t want to spend time chasing down one million references no one reading this blog is likely to follow up on, but this is a rich area of inquiry with many useful applications, especially for predictive modelling. In particular, deep learning has some great potential to assist medical clinicians with diagnosing medical conditions that rely on imaging like cancer detection.

0.1 Overview of the Problem

The mixed model situation in R is a hot mess (that is, it is spectacularly disordered).

The linear mixed model ecosystem in R consists of over 80 libraries that either construct and solve mixed model equations or helper packages the process the results from mixed model analysis (this list was derived from a similar list created by Ben Bolker and others). These libraries provide a patchwork of overlapping and unique functionality regarding the fundamental structure of mixed models: allowable distributions, nested and crossed random effects, heterogeneous error structures and other facets. No single library has all possible functionality enabled.

This patchwork of packages makes it very challenging for statisticians to conduct mixed model analysis and to teach others how to run mixed models in R. As a statistical consultant, my job is to help agricultural researchers develop statistical models that match their research aims and implement those models in R or another statistical software. My experience after 4 year of consulting is that many researchers have a hard time time sorting out how to correctly specify their statistical model in R.

Please note that none of this is intended as a criticism of current or past package authors. Package authors and maintainers give a tremendous amount of time and effort - often for free - to provide the functionality present in a package. It is not the fault of any individual that the R mixed model ecosystem is so challenging. It is likely more a reflection of how challenging mixed models are to properly specify and implement and the overall flexibility of these models.

But alas, here we are. Let’s explore the specific problems with running mixed models in R.

0.2 The Issues

(arranged in order how of much grief I think each causes for R users)

lme4::lmer(y ~ x + (1|rep), data = mydata,  
        na.action = na.exclude, REML = TRUE)

nlme::lme(y ~ x, random = ~ 1|rep, data = mydata, 
       na.action = na.exclude, method = “REML”)

asreml::asreml(y ~ x, random = ~ rep, data = mydata, 
              na.method.X = 'exclude', 
             na.method.Y = ‘exclude’)

rrBLUP::mixed.solve(y = mydata$y, x = {fixed_design_matrix}, 
                    z = {random_design_matrix}, SE = TRUE)

0.3 Existing Solutions

There have been recent efforts to create a common function call for mixed models with the library multilevelmod (part of the Tidymodels family) and a common way to access accessory information with broom.mixed. These umbrella libraries are enormously helpful, but they are incomplete, addressing less than 10 mixed models packages in total.

0.4 What else Is needed?

Unfortunately none of these solutions solve one of the most bedeviling problems: running a complex mixed model in R requires using a large patchwork of packages (many of which are not part of Tidymodels) and in some cases, these different functionalities cannot be implemented together.

0.5 What else can we do?

• Teach Tidymodels to our clients. I admit to not doing this, but I should be doing this.
• Encourage teaching of Tidymodels framework at our institutions.
• Write an opinionated white paper about recommended functionality and style for any new mixed modelling packages written for R (Who wants more things to do??? And will anyone even read this, let alone follow it?)
• Contribute to multilevelmod and broom.mixed to expand the packages supported. Important Note: doing this starts with visiting the respective Github repo’s for these packages and finding out what the package authors & maintainers want.

Questions to ask ourselves:

1. Which is the most important barriers we should address: standardized syntax? standardized output? something else?
2. What mixed model packages are the most important to the user base?
   
3. Seriously, who is going to do this? Is this grant proposal worthy?
   
4. What about the functionality patchwork problem? What on this green earth should we do about that?

Other ideas? (I want to hear them! DM me on Twitter: @SeedsAndBreeds)

0.6 Final Thoughts

I decided to write this after years of trying to make mixed models work for my clients and often seeing them frustrated. Perhaps our applications seem niche - but, there are thousands of agricultural researchers in the world. And questions about how to build a linear mixed model and conduct ANOVA are by far the most common FAQ I encounter. Common scenarios in this subject include:

1. A multilevel field trial (imagine 3-5 treatments, each with 2-10 levels, perhaps one a time series, because ag scientists love those complex experiments!). In many cases, we need to account for spatial variation.
2. A field trial with one fixed effect and one random effect, but two different heterogeneous covariance matrices need to be included: one for spatial variation and another for relatedness among the treatments.
3. All of the previous experiments plus special designs: split plot, split-split plot, Latin square, incomplete block, augmented design….
4. All the previous experiments, but now we have a beta-distributed variable! Or a negative binomial (the experimental variables I see rarely meet the expectations of a Poisson distribution).
5. A mixed model with tons of zeros! And maybe a time series component, as well.

There are ways to solve these problems–easily–in other (paid) software such as SAS or asreml-R. But for whatever reasons that I am not in control of, people want to use R and the non-proprietary libraries that accompany it. And often they end up implementing sub-optimal solutions based on a lack of understanding of how to properly specify a mixed model in R.

Should we abandon users to the confusing and incomplete maze of mixed models in open-source R or should we help them and ultimately create a more sustainable mixed model ecosystem?

the R mixed model ecosystem https://t.co/wx5nLM1soc

— ag stats stan (@SeedsAndBreeds) July 3, 2022

A video of this talk can be viewed on youtube.

The Republic:

The Jedi maintained decent, well-organised archives in a large library and they even had paid librarians on staff. But any Jedi can delete information and no record is left regarding this? For heaven’s sake, an entire planet was removed from said archives That ain’t good. It led to the Clone wars and eventual rise of the Galactic Empire. It also turns out that a person just has to look like a Jedi in order to steal secured jedi archives.

Original Galatic Empire:*

This is where all the bad stuff starts. Initially, they had a good system for managing their vasts amount of data and an off-site backup with a very powerful firewall. However, once that firewall was breached, there was poor security inside - I mean, a dead archivist’s hand could be used to access the entire system, presumably as a root user. Their storage facility also relied on a rather inconvenient system for accessing data: a massive tower than required a remote-controlled tool for physical retrieval of data files.

The tool itself required manual operation and apparently the entire facility had no backup generator to maintain security and functioning in case of a power loss. Incredibly, there was also no search tool. Data archives had to be browsed (what???). it’s really quite astonishing that anyone ever found what they needed. Options for sending out the data were also remarkably cumbersome - a satellite dish the size of a Star Destroyer has to manually turned in the correct direction to send a file out (was the Empire capable of automating anything?). I also wonder about the file compression capabilities - seriously, is there any file so large it needs a satellite dish of that size?

The First Order:

The First Order inherits a number of problems from the original Galactic Empire, most notably insufficient backups resulting in lost information. A map to Luke Skywalker is lost, making it nigh impossible to find him. A map to Exogol had two copies ever made, one of them on a destroyed death star sitting in the middle of raging ocean? It’s amazing these were very found. Without these maps, we have no Sith and no Jedi (maybe not such a bad thing).

The point is that if the Jedi had better data practices, then they might have avoided the rise of the empire. And perhaps if the Empire had better data practices, they may not undergone such a rapid downfall.

What happens with good data management:

• Secure data that can only be modified by authorised users
• Validation of data values to prevent errors
• Data is accessible: it can be accessed in many places, by many, in formats that are easy to use
• Stored data is easy to search and summarise
• There are sufficient backups if something goes wrong

Wait! This actually matters!

This is more than a story to amuse folks, though. Poor data management is a rampant issue across academia. I have observed that most researchers do not document their data processing in sufficient detail to recreate the process. Who else has read this statement “outliers were removed” with no details into how those outliers were eliminated or the numerical values of those “outliers”?

From a research standpoint this means the analytical procedure is not replicable. That means someone starting with the same raw data will not obtain the same results following the described analytical procedure – because the description was inadequate. This can also mean data are unreliable. If the original version of data become lost as individuals clean up the data and do not document their process, then there is no way to evaluate if the cleaning was correctly done or justified. Data loss is always a risk when files are emailed around rather than shared via a more permanent data storage solution. All of these risks become amplified when an individual leaves a research program and can no longer be reached to resolve data questions.

Tools for better data management

Handily, many many tools exist to assist in data management. I give a bare bones outline in this post (please read this!!). Software Carpentry provides an excellent outline of essential processes in their document Good Enough Practices for Scientific Computing.

Below is an even more bare ‘bare bones’ outline.

One of the best solutions is to put your data in a database. This a great tool for large quantities of data that follow a regular format and are frequently updated - such as variety testing data.

However, most agricultural researchers are not so lucky to have such structured data. At the very least, maintaining research outputs in a shared and secured location is a good start. Archiving raw versions of data is paramount. This maybe never be needed, but if it is needed, there is no replacement if those data are lost. Requiring complete documentation from all research contributors on data set generation and processing is also good. A “codebook” is another great tool. This is a file accompanying a data set that provides the name of each column, what data are present in that column, and optionally, the units of those data and the range of values expected. If a column contains contains numeric values between 0 and 100, then others will know that 980 is probably an error.

Most of the researchers I work with were trained in a different era - when data was expensive to obtain and as a result, there was considerably less data to deal with. I count myself among this group. At that time, data management tools were less relevant because we had the time to comprehensively examine a data set for errors. Furthermore, data sharing and data reuse were not widely done. But, we all now those days are long over. We are now expected to process a large quantity of data rapidly, analyse it and eventually archive it publicly for possible reuse by someone else. Engaging in practices that enable quality data management is essential for participation in today’s research environment.

Sources

Thomas Lumley, Github repo useRfasteR
Hadley Wickham, Profiling, Advanced R
Dirk Eddelbuettel, Rcpp

The Process for Improving Code:

(quote from Advanced R)

1. Find the biggest bottleneck (the slowest part of your code).
2. Try to eliminate it (you may not succeed but that’s ok).
3. Repeat until your code is “fast enough.”

Easy peasy, right???

Some general guidelines for speeding up R code

1. Use data frames less - they are expensive to create, often copied in whole when modified, and their rownames attribute can really slow things down.
2. Be wary of using functions that copy objects in whole: c(), append(), cbind(), rbind(), or paste(). When used in loops, you can get a massive proliferation of objects in memory.
3. Use vectorised functions:
   • apply, lapply, sapply, vapply, tapply, mapply
   • rowSums, colSums, rowMeans, colMeans, cumsum, diff
4. Base functions are designed to handle wildly different input. Consider rewriting base functions for highly repetitive tasks.
5. Use parallel::mclapply for parallelising functions.
6. Consider an optimized matrix algebra library (beyond BLAS) for better performance (e.g. Apple vecLib BLAS, openBLAS).
7. If you work with sparse matrices, use tools for them like the package ‘Matrix’.
   
8. For huge objects, consider storing the information in a database and accessing it with ‘dbplyr’. The packages ‘dbglm’ and ‘tidypredict’ will also do model fitting with data inside a database.
9. Another solution for large objects are specialized formats like HDF5 or netCDF.
10. Take advantage of the Rcpp suite of programs - not just for C/C++ programmers (e.g. RcppArmadillo::fastlm).
11. Use an alternative implementation of R (e.g., fastR, pqR).
12. Check your code with benchmarking!!!

Let’s do some benchmarking!

Important: don’t (re)install ‘compiler’; you should just be able to load it in R v3.5 and later.

pck <- c("pryr","microbenchmark", "profvis", "compiler", "mnormt")
invisible(lapply(pck, library, character.only = T))

First, Turn off the just-in-time compiler. Note that return value is what the JIT was set at previously (default = 3).

enableJIT(0)

a <- function() {
  m <- sample(1:100, 2)
  data.x <- lapply(m, function(x) rnorm(1e4, mean = x))
  do.call("cbind", data.x)
}

some_data <- a()
dim(some_data)

microbenchmark(
  mean_loop = apply(some_data, 2, mean),
  mean_vec = colMeans(some_data),
  mean_manual = apply(some_data, 2, function(x) sum(x)/length(x)),
  mean_manual_ultra = apply(some_data, 2, function(x){
    total = 0
    n = 0
    i = 1
    while(!is.na(x[i])) {
      total = total + x[i]
      n = n+1
      i = i+1
    }
    total/n
  })
)

methods(mean)

x1 <- list(e2 = runif(1e2), e4 = runif(1e4), e6 = runif(1e6))

lapply(x1, function(x)
  microbenchmark(
    mean(x),
    mean.default(x)
  )
)

# tracking package type, etc
otype(mean)
ftype(mean)
showMethods(mean) #for S4
methods(mean)

methods(var)
getAnywhere(var)

Rprof("permute.prof", memory.profiling = T)

sigma.mv <- diag(1, nrow = 5, ncol = 5)
sigma.mv[upper.tri(sigma.mv)] = 0.5
sigma.mv[lower.tri(sigma.mv)] = 0.5

mvn.data <- rmnorm(1e3, mean = rep(0, 5), varcov = sigma.mv)
colnames(mvn.data) <- c(paste0("x",1:5))

kmeans.Ftest <- function(kmean_obj) {
  df.1 = length(kmean_obj$size) - 1
  df.2 = length(kmean_obj$cluster) - length(kmean_obj$size)
  betw_ms <- kmean_obj$tot.withinss/df.1
  with_ms <- kmean_obj$betweenss/df.2
  fratio = betw_ms/with_ms
  pval <- pf(fratio, df1 = df.2, df2 = df.1, lower.tail = F)
  stuff = c(fratio, df.1, df.2, pval)
  names(stuff) <- c('F-ratio', 'df 1','df 2', 'p-value')
  return(stuff)
}

kmeans.optimiz <- lapply(2:10, function(x) {
    results = kmeans(mvn.data, centers = x, nstart = 15, algorithm = "MacQueen",
                        iter.max = 50)
    kmeans.Ftest(results)
})

kmeans.final <- do.call("rbind", kmeans.optimiz)

Rprof(NULL)
summaryRprof("permute.prof") #, memory = "both")

p <- profvis({
mean_loop = apply(some_data, 2, mean)
mean_vec = colMeans(some_data)
mean_manual = apply(some_data, 2, function(x) sum(x)/length(x))
mean_manual_ultra = apply(some_data, 2, function(x){
  total = 0
  n = 0
  i = 1
  while(!is.na(x[i])) {
      total = total + x[i]
      n = n+1
      i = i+1
    }
    total/n
  })
})
htmlwidgets::saveWidget(p, "profile.html")

browseURL("profile.html")

svd
La.svd

try(show_c_source(.Internal(La_svd(x))))
show_c_source(.Internal(mean(x)))

download.packages("broman", repos = "https://cloud.r-project.org",
  destdir = getwd(),type = "source")

a <- letters[sample(10)]
tracemem(a)
a[1] <- "Z"
b <- a[1:5]  # not copied

mem_used()
object.size(x1) #base
object_size(x1) #pryr
compare_size(x1) #between base R and pryr

enableJIT(3)

0.1 Remember!

xckd comic