statistics -- some foundation

Statistics overlaps into Math of course. We can record samples into a CSV or spreadsheet and do histograms, and (Algebra 1) regressions. Yet CSV data are arrays. Arrays are matrices, and thus vector, evaluable (Calc 3), as well as Python and R friendly. IMO, because it spans both simple and complex Math, and CS, Stats are a fun side project during one's Math progression, at any time.

Basics are Algebraic if one uses tables for area under curves. If not, a person could do them using simple Calc 1 calculations. TI-84's -- if available -- are also great for checking that kind of calculation (eliminating tables and Calc) and for offloading the drudgery of long lists.

HS AP Stats is essentially an Algebra-only college Stats1 class, and slightly more user friendly. A review book for AP Stats, the Princeton Review, Table of Contents is a list of roughly 35 subjects, split over 4 categories. I've used it below with some modifications to describe what a person might want. Beyond basic or AP, I've added a "next level" section at the bottom, since matrices connect Math to Statistics and arrays/data structures connect Stats to Data Science.

Introductory data data collection tabular methods graphical methods - qualitative graphical methods -quantitative numerical methods - continuous boxplots add/mult a constant comparing mult. groups bivariate data: covar,regression categorical frequency Sampling and experiments plan a study data collection plan a survey bias in surveys plan an experiment Anticipating outcomes probability

random variables probability dist. - discrete random var's probability dist. - continuous random var's normal distribution combining independent random var's sampling distributions Statistical inference confirming models parameters point estimation interval estimation confidence interval inference: significance tests hypothesis: testing and accepting estimation and inference: population proportion estimation and inference: population mean estimation and inference: 2 population proportions estimation and inference: 2 population means inference: categorical data

For a comprehensive course, there's Khan Academy. On YouTube, I prefer Brandon Foltz, M.Ed video compilation, esp at 0.5x speed. They were made 2010-ish, but ahead of their YouTube time¹. Below there are some from Foltz and from many great teachers or vids which include nuggets in some way. As one goes along in review, it's a chance to relearn the annoying Greek symbols for parameters. The same Greeks as in finance, but different usage.

¹Another brilliant info-sharing hero with early YT chops was Derek Banas. One of his best might be his investing video, which crosses data science with financials and Python. And of course Barry Brown for programming any of it in C.

---

distributions of data obtained

degrees of freedom, clt

To Z or t (38:16) Brandon Foltz, 2012. The notion behind this choice, without calculation. Almost all stats are samples, not entire population, ie, "the prevalence of depression of those over 65" - how to give all questionnaire? Can't. The smaller the number, the more chance of error. Avg temp in NYC, what if we just used a day, or 5 days? The larger the sample the more confidence we have captured reality.16:00, 33:00 decision. 19:00,30:00 degrees of freedom
Buying land (32:06) Brantley Blended, 2018. listing, PLAT recency, survey recency (due diligence period), boundaries,

bivariate data: linear regressions, covar

covariance (5:55) Ben Lambert, 2013. intuition behind covariance. Positive, negative, and none. Some formulas, but conceptual.
Variance vs. covariance (Webpage) Investopedia.

combinatorics, permutations, probability

nCr and nPr. These are the denominators of probability, since they give the universe by which we determine our odds. If "statistics" are the collected data, nested inside we have probability (based on prior events, statistics), and inside this is the "combinatorics and permuatations" which create the denominators of probability fractions.

Permutations and Combinations (17:40) Organic Chemistry Tutor, 2017. Simple review of when to use nCr, nPr, or just the factorial by itself. 14:00 How many ways can we arrange the letters in the word "Alabama".
problem - probablility (replace/don't replace) (10:11) Amy Krusemark, 2020. a slight political note at start, AP question. example of non-replacement probability, example of understanding permutation/combinatorics effecting denominator. 2:40 without other notice, 0.05 alpha is threshold for a reason to doubt.

---

next level

KL Divergence

KL Divergence (18:13) ritvikmath, 2023. Non-negative comparison of two distributions.

vectors and matrices in statistics

As my friend Bart texted...

Matrices are really just notation for a list [of] equations. Not so profound, not for a long time. Like if I have ten data points then i have that equation ten times. Ten y values. Ten x1 values, ten x2 values, etc. Thirty diff x values, so x could be written as a 10x3 grid. Ok enough [of] that
Beta is 1x3 and and x is 3x10. Then when multiplied out gives a 1x10 vector. To say that vector equals the 1x10 y vector is to say each component of y equals the corresponding for RHS. ie ten equations

And of course wherever we have a matrix, we have a potential vector.

Comp Sci v. Data Sci Matrix (10:34) ritvikmath, 2019. Reveals some clear differences between computer matrix use and math/datasci use and how they overlap (eg. CompSci efficiency can work on any math app).

data structure

necessary for data science. Data Science will evaluate these further, sometimes using Calculus, but we at least need to know what they are, IMO. Not on the AP test. The terminology transform is Statistics and Math use the term "matrix", but Data science/Computer science use the term "array". Depending on what we need to do with the matrix, computing will perform some function on a data array. Usage: imagine an R2 scatter plot, but where we have a third dimension with error information attached to each data point.

Linear Algebra: Transformational Matrices Part I (15:43) Computer Science, 2021. transformations on R2 matrices, using geometric examples. There's an entire playlist that's valuable.
Linear Algebra: Transformational Matrices Part II (9:23) Computer Science , 2021. transformations on R3 matrices, as we might do with data structures.
Trig Functions (9:15) PatrickJMT, 2011. this hero scores yet again. Just in case you need them again for the stuff above. So rotten.
1,2, and 3d structures (8:32) GridoWit, 2017. basic terminology and location tracking within different types of arrays and data structures. The inuition that arrays solve a storage problem: we don't want to have a new variable for each piece of data we have collected. C syntax is also provided.

statistics -- simplistic error catching in studies

Note: I've also included some statistics-relevant information from my 2020 Maths post within a "resources" section at the bottom of this post.

This list so far only includes traditional "p-value" errors, and not Bayesian style errors. But when we look at published papers in Nature or at the NCIB, what are some rule-of-thumb errors we can look for?

short list

• correct test for the hypothesis
  Look for a hypothesis to be clearly stated and that the null has been rejected. That said, having pre-determined hypothesis and the requirement for rejection of null is an outdated method to some: a more modern approach is to crunch an immense amount of data through arrays and let the data itself identify anomalies from which to draw conclusions. However, this method is still used in many studies required by, eg. the FDA and many scientific papers.

  Types of tests (19:53) Amour Learning, 2019. Generally t tests make less presumptions than Z tests and are better. 12 main types of tests and when to use for which hypothesis.
  Selecting the correct test (12:31) Erich Goldstein, 2015. What type of data (categorical or quantitative)

• a priori standards
  Is the alpha value determined prior to the study and clearly stated? A typical alpha value of 0.05, looks for a 1 in 20 chance the results would happen by chance, and obviously an alpha value of 0.10 that 1 in 10. NCBI:nih.gov
  Counter: the most common reduction is for a third variable. I'd like to propose another: studies designed with an a priori alpha presuppose the a priori hypothesis on which is reliable. However, data gathered in the study may reveal other information. There may be some statistical significance for although it's not clinical.
• publication bias
  My friend Bart notes

  If ten people examine a question and none of them do any datamining and only the people who happen to get a positive result submit or publish, then the stats seem ten times as strong as they are.

• randomization problems
  Front-end randomization is a thing, but there are other error possibilities later in the study also, well-described here.
• control group
  Perhaps too obvious to include, but included anyway

resources

p value/alpha value

Only Bayesian statistics can prove theories, but P value stuff can prove data. It's a lower standard than Bayesian but useful, of course.

What's a P value (20:30) Daniel Lakens, 2019. some accent -- good at 0.75x. Good backgrounder. No math. Notes Bayesian necessary on theories. 8:45, otherwise all we can do is rule out something (using P).
Baye's theorem (15:10) 3Blue1Brown, 2019. new information, and how to use in research. Graphically illustrated.

12 main types of hypothesis tests

As noted above, they must be matched to the hypothesis, but what are they?

concept instruction

The clearest statistics instruction videos for my time are from Brandon Foltz. "Only" an M.Ed (probably now a Phd), found ways to create clean graphics, and years before clean graphics were common. Another excellent early adopter, although only in an overlapping sense with statistics, is Derek Banas. One of his best might be his more recent comprehensive portolio using Python. I believe Derek had a job with Apple when he was 16.

spreadsheets, Python, R, SPSS

Most people cannot afford SPSS, ergo PSPP is a a good option. If using Linux, most repo's have it. In Arch, it's in the AUR. There's plenty of PSPP videos on YT, as well as SPSS videos that a person can adapt.

This guy's series is *incalculably* helpful, har har. He as MSoft AF, Vstudio, Excel, Jupyter Notebook, but it's just as easy to do all of his stuff in Google Colab with Google Sheets as-is, or download them as CSV's first and then merge them.

Pandas Merge (Excel) (9:08) Alan Hettinger, 2022. Similar to a SQL join. When we want to put things together from various tables into a combined output.
Pandas Merge (Sheets) (10:48) Tobias Willman, 2020. Adapt the Jupyter above using this guy's vid.

database options

Right tool for the right job. An easy case might be a password database, essentially a flat file database, Excel 2-D type of thing. A more difficult case is a video edit. You've got clips, which might expand in number, you've got recipes or methods for various transitions and combinations, and you've got an overall project view which pulls from these. Whatever we have we'd like it to be ACID compliant. Try to use umlet or some such to do the UML.

7 Database Paradigms (9:52) Fireship, 2020. Seven different ways of looking at data, with suggested applications.
issues univ of utah (19:49) Mark Durham, 2018. Has a great chart.

relational algebra, calculus

Relational Algebra is just a form of logical operators (most are shown below), easy to understand for those who've taken logic courses.

Linguistically, Relational Algebra imparted its first name, "relational", to databases constructed around its principles, ergo "relational databases". In other words, relational databases are databases built around Relational Algebra.

Relational Algebra itself was created by Edgar Codd. His main book on this is The Relational Model for Database Management, and his Turing Award paper on relational algebra (1981) is available online (PDF). The most important skill level to reach is the capacity for fast database normalizing. This takes practice, similar to the logic skill for taking statements, text or otherwise, and symbolizing.

Relational Algebra Lecture 1 (50:21) Database Psu, 2015. Penn State. The first in their amazing free online course. Lecture begins about 19:12.
Relational Algebra Pt 3 lecture (51:18) Database Psu, 2015. The various operators and their uses.

Relational calculus is a way of making queries using relational algebra to get what we want from our queries. It's less atomic than relational algebra.

relational keys (primary, foreign, composite, blank)

Link: postgres constraints ::

Relationals are a bit of a pain. See my later post on how to configure PostgreSQL for local access. Restrictions make searches easier. The main idea is a unique column (Cust ID), can then access a unique row in the next table. In the pic below, the primary is green, the foreign red. A column with a "references constraint" on it is a foreign key.

Primary and foreign keys (2:08) Udacity, 2015. Primary and foreign keys explained. Foreign key is column with references constraint. No lookup table is used, but it might be possible by indexing somehow.
Primary and foreign keys (5:24) Prescott Computer Guy, 2011. This has an intermediate lookup table. He wants every table to have an ID column, to be sure to have a unique value for that table.
Postgres version (5:42) Trent Jones, 2020. Unfortunately uses PgAdmin not DBeaver.
SQL from teaching perspective (10:12) Mike Dane, 2017. Exam configuration

• primary key: Postgres/SQL constraint equivalent - NOTNULL, UNIQUE
• foreign key: Postgres/SQL constraint equivalent - none. Verifies that a column from keyed table always has a value in another table (is linked).

constraints
PostgreSQL	MySQL
PRIMARY KEY	PRIMARY KEY
FOREIGN KEY	FOREIGN KEY
UNIQUE (column)	UNIQUE
NOT NULL/NULL (null allowed)	NOT NULL
CHECK	CHECK
DEFAULT	DEFAULT
CREATE INDEX	CREATE INDEX
EXCLUDE

start-up/shutdown (postgresql)

See my other post for how to configure as user. That done, it's a pretty simple....

$ postgres -D '/home/foo/postgres/data'

With the server is up and running, you can open another terminal and run psql commands. If you run as user postgres (see below), you'll have admin authority.

$ psql postgres

If a lot of data has been written, a good exit phrase to make sure all is safely written:

$ psql -c CHECKPOINT && pg_ctl stop -m fast

bonds

Dealing in bonds is a rich universe of calculations and ratings. Within such an environment, a large terminology set is expected. One can go in-depth and attempt to remember the vocabulary. IME, an easier start on useful bond terminology is to know the six elements we use for pricing a bond. Once pricing, the difficult kernel, is understood, the peripheral terms became clearer to me. The main pricing terms are shown in the graphic below, only lacking the "market price". The market price is another name for a type of benchmark, a concept slightly more complex than can be explained in the graphic anyway.

"Coupon rate" is also sometimes called the "contract rate". It's the annual interest rate the issuer pays on the bond. Accordingly, a "zero-coupon" bond pays no interest.

primary or secondary

With these elements, how does one determine how to purchase a bond? Unlike the equity market, we can calculate what to offer for a bond on any given day. This bond price will be moving if we're looking at the secondary market.

• primary market: revenues from the bonds go directly to the issuing organization as capital. Typically, these are sold at par.
• secondary market: the issuing organization receives no proceeds -- these are sales between customers. Prices fluctuate by the second, same as equities. The categories are discount, par, and premium.

discount, par, premium

The results of our price calculation will result in the bond's current price having a designation: discount, par, or premium, depending on how the price corresponds to the face value of the bond.

maths resources

Links: Photomath :: Wolfram Alpha :: Desmos
xda xiaomi redmi2 :: xiaomi community :: more flashboot commands :: reddit post (self-teach calc) :: Ranking engineering degrees by difficulty (15:07), 2023

---

Math is difficult, because the explanations come too late in the game. You have to get to Linear Algebra to explain basics, which most people can't accomplish

spreadsheets, Python, R, SPSS

Most people cannot afford SPSS, ergo PSPP is a a good option. If using Linux, most repo's have it. In Arch, it's in the AUR. There's plenty of PSPP videos on YT, as well as SPSS videos that a person can adapt.

Python, Numpy, Plotly (1:10:57) Derek Banas, 2022. Uses Jupyter on Anaconda but a couple tweaks will do it in Google Cloud. 16:00 explains Numpy, 17:00 read CSV, scraping 21:00, Plotly 45:00
Pandas Merge (Excel) (9:08) Alan Hettinger, 2022. Similar to a SQL join. When we want to put things together from various tables into a combined output. Can dapt Jupyter to Google Colab with Google Sheets as-is, or download them as CSV's first and then merge.
Pandas Merge (Sheets) (10:48) Tobias Willman, 2020. Adapt the Jupyter above using this guy's vid.

Self-study generally

self-study feynman's books (27:33) Math Sorcerer, 2023. Mathematics for self-study series of books, espec trig and calculus.
V600 scanning tips (5:53) erwnalejo, 2021. Somewhere in Western Kansas. Lomo digitaliza, blue tack to adjust height. Links in comments to products. Lomo is an extra $50.
epson scanning tips (28:11) Nick Carver, 2019. Paper masks some of the holder. 5:40 uses a squeeze ball. 2400 dpi is fine for any normal use including 8.5 x 11 blowups, etc.
V600 overview (52:17) Film Friends,

PreCalc

Two best books I know are Larson and Sullivan's. I found a 9th edition Sullivan and a

Polar coordinates

Difficult because we move back and forth between analytic trig and graphic trig, then introduce time changes in the angles

Series

Infinite series means infinite terms, not that the sum diverges to infinity. Also the best way to learn them is M1 in MVP math.

Harmonic Series (sum of sums) (46:34) Mathologer, 2020. Infinite sum of 1/(n+1) Gamma explained at 24:21.
Buying land (32:06) Brantley Blended, 2018. listing, PLAT recency, survey recency (due diligence period), boundaries,

Matrices

My road to matrices has been difficult, b/c for years it was one of those categories where 1) I simply had to remember formulas and 2) Math teachers acted like matrices were easy, like I was an idiot, because the problems were easy. Nearly always in Math, my intuition is right when something doesn't quite seem that easy. So lets call matrices what they are: representations of vectors with several degrees of freedom, not just 'simple arrays'. Although the latter is correct, it's like saying cars are easy to make b/c they fit in a garage.

The real way to look at it is via transformations, rotations, and vectors, as well as ways to write functions (remember we can also write functions with limits). That is to say, linear algebra. M

1. Berkeley HS Math 1, Module 6 (webpage). This has some good videos on different transformations: reflections, rotations, and translations. The shortcoming is that the they move the points, instead of changing the underlying grid (as we do with matrix scalars).
2. Linear transformations (10:58) 3 Blue 1 Brown, 2016. The real value of what matrices actually are, and also acknowledged by the host. They know how important this is. Connecting matrices and vectors. 3:52 i-hat (x), j-hat (y).
3. Why can't we multiply vectors (51:15) Freya Holmer. A good review, awkwardly stated, of basic maths up to matrices/vectors. Multiplying R3 vectors gives quaternions, multiplying R2 vectors gives us complex numbers.
A tour of differential equations(27:16) 3 Blue, 1 Brown, 2017. How we work backwards with diff equations
4. interesting vector usage (14:40) Kieran Borovic, 2023. an interesting way to convert categories of nature into a 7 row vector.

Integral tricks

Most difficult are problems that combine U-sub (reverse chain) and IBP, integration by parts, but also must know d/dx, dx/dy, Dx

Integral conceptualization (10:54) Math The World, 2024. *does not* use the best Cherkassky/Russian way of series, however gets much nearer than most and points out the problem of focusing too much on area under curve.
Why dx at end of integral (4:36) Krista King, 2016. sometimes I forget why it's smashed onto the end of an integral or double integral and so this is a good reminder it's the dx(limit of distance,0), not the Dx (distance).
What is dx alone? (5:38) Jim Fowler, (2013). Just as important as the one above. A little deeper.
u-subs (11:02) The Organic Chemistry Tutor, 2018 . a few definite integrals from organic chemistry tutor.
Cheat sheet (PDF) Most of the tricks and common integrals.

Exponents/Logs

exponentiate (30:08) Mathemaniac, 2024.

Lesser used

I consider that these are like the irregular verbs of math. We have to know them typically by use, as they are difficult to remember how to derive. Trig identities, hyperbolic derivatives and integrals. Geometric Algebra -- nuclear physics.

Logic

Russell's paradox (28:27) Jeffrey Kaplan, 2022. Covers naive set theory inadvertently (as opposed to axiomatic set theory).
attach/remove ft lens (2:46) Caleb Ginsburg, 2021. Turn to f5.6, then turn speed selector to 400 ASA in alignment with 5.6, then push the release button. It sort of hinges from top to bottom.

Statistics

Old skool brandon foltz, are still some of the best videos out there.

the most important statistical skill (13:35) Very Normal, 2024. Monte carlo is about iterations

Is an ellipse really a conic?

I don't think so. Consider that an ellipse is supposed to have equal eccentricity on each end -- it's not supposed to be shaped like a cross section of an egg.

But now consider that a cone has a varying radius from it's base to its vertex. At the vertex of a cone, the peak of the cone, the radius is zero, at its base, the radius of the cone is however large you want to make it.

The ellipse is (supposedly) the result of cutting the cone at a SLANT. This means the radius of the cone is different at one end of the ellipse than the other. This means that the only way to have a true ellipse is to have a section of a cylinder, not of a cone. Nevertheless, they persist in calling it a "conic". Is there something I overlooked?

Math - Grammar - Law

Geometry tends to bore people, for example, me. It might be interesting in one way though. There's a connection we feel between Math, Grammar, and Law (within this essay, "MGL") that emerges when we do Geometric proofs. I don't think there's any other subject where there's a convergence like this. What allows them to work together appears to be the rule structures underlying each concept.

Familiarity, intuitive use
Most of us who are not professionals in MGL fields still operate with them enough to have a feel for them. In Math most of us recall that different theories seem to apply to different types of problems, for example, commutativity and additive inverses. Without knowing the names of the rules, we understand that 3 - 2 is not the same as 2 - 3, but that 2 + 3 and 3 + 2 are equal. In Grammar, we know that adverbs describe actions, but even if we've forgotten the name of that rule, it feels wrong if we mistakenly use an adverb to describe a noun. Eg, "An unused heavy weight makes a good doorstop." looks correct, as compared to "An unused heavily weight makes a good doorstop". In Law, most of us understand that, if contracts are broken, there might be a lawsuit, whether or not we happen to know the applicable law. So in each of these areas, most of us operate out of habit without needing to consult the specific MGL rule that applies. In Geometry, when we make proofs, we have to be explicit about only the Math portion, but there are other laws at work.

Grammmar
Grammar is determined by usage and social convention. In the case of US English, media status and academia tend to promote one form of usage into Standard English and Received Pronunciation. These then normatively reinforce that usage over others. Outside of this politicking however, grammar "rules" are descriptive, not normative. they are descriptions of what we observe across languages. One of the potential hiccups is agreeing on the linguistic terms. Linguistic terms are themselves words -- defined partly by usage -- so that they risk a circularity of using themselves to define themselves. We break the circle fairly effectively by first having a conversation about what our linguistic terms refer to, a meta-conversation, attempting to solidify what our discussion will point out in a language before we start examining languages. Outside of Linguistics study, in the world. language operates without bounds. Inside Linguistics, that is, while studying worldly language effects, we want own words to point to agreed-upon concepts.

As an example, let's suppose we're Linguists who agree about the meaning of the word "case", insofar as language is concerned. Using this definition, we're able to observe the number or nature of cases across languages in way we both understand. In German, we could agree there are likely at least 4 cases; the nominative, accusative, genitive, and dative cases. Observing English, we might agree that typical English usage does not split these apart so neatly. The language rules society follows come through usage, the rules we use to describe them must come through academic agreement. In one sense, Grammar is a posteriori, but the study of Grammar requires a priori definitions.

Math
With Math, we believe we establish rules based on logic first, and experience second. So, the trajectory is deductive, we start with principles and build logically from that point to conclusions within that logic, or must expand it. The basis is a priori, but the expansion of axioms to encompass new information is a posteriori. Of course, Kant considered Mathematics "synthetic" for this blend. But what about our meta-conversation in Math, like the one we had in Linguistics? Isn't it true that we must first have a conversation about what Math words mean and agree upon what they point to, before we subsequently do that Math? For the part of Math that is arithmetic, that uses symbols, such as 2+2=4, it's relatively easy to agree because there are quantities these point to, not just concepts. We can place two items in front of anyone in any language and, once it's clear we are discussing a quantity, need only agree on the symbol. So for quantitative Math, there can be agreement about the language. But Math is not all quantities, it's also performing operations on quantities and applying theorems to those quantities, and these operations and concepts may need clearly agreed terms to discuss.

Law
With Law, the statuatory scenario is a priori; the law is true by its definition as declared, not through observation (inductively). In this way it is like the first discussion we had about Linguistics, we had to first agree what we meant by the words we were going to use with the However, with precedents, the law also allows for interpretation which can affect its implementation in an a posteriori sense.

Geometry connection
One thing that seems to combine all three of these is Geometry. Geometric proofs utilize deductive logic, the obviousness of quantities and graphical constructs, and clear definitions. Perhaps Geometry is not so boring after all.