List of boxes

Contents

Dedication

Acknowledgements

Chapter 1 - Introduction

The Problem and Objectives

Outline of the Book

Chapter 2 - Rational Inference: A Constrained Optimization Framework

Inference under Limited Information

Qualitative Arguments for Rational Inference

Probability Distributions: The Object of Interest

Constrained Optimization: A Preliminary Formulation

The Basic Questions

Motivating Axioms for Inference under Limited Information

Axioms Set A: Defined on the Decision Function

Axioms Set B: Defined on the Inference Itself

Axioms Set C: Defined on the Inference Itself

Axioms Set D: Symmetry

Inference for Repeated Experiments

Axioms vs. Properties

Summary

Appendix 2A: Axioms Set B - A Concise Specification

Notes

Exercises and Problems

Chapter 3 - The Metrics of Info-Metrics

Information, Probabilities and Entropy

Information Fundamentals

Information and Probabilities

Information and Entropy

Information Gain and Multiple Information Sources

Basic Relationships

Entropy and the Grouping Property

Relative Entropy

Mutual Information

Axioms and Properties

Shannon's Axioms

Properties

Summary

Appendix 3A: Wiener's Derivation of Information

Notes

Exercises and Problems

Chapter 4 - Entropy Maximization

Formulation and Solution: The Basic Framework

Information, Model, and Solution: The Linear Constraints Case

Model Specification

The Method of Lagrange Multipliers: A Simple Derivation

Information, Model, and Solution: The Generalized Constraints Case

Basic Properties of the Maximal Entropy Distribution

Discussion

Uniformity, Uncertainty, and the Solution

Conjugate Variables

Lagrange Multipliers and Information

The Concentrated Framework

Examples in an Ideal Setting

Geometric Moment Information

Arithmetic Moment Information

Joint Scale and Scale Free Moment Information

Likelihood, Information, and Maximum Entropy - A Qualitative Discussion

Summary

Appendix 4A: Uniqueness, Convexity, and Covariance Uniqueness Convexity of the Normalization Function and the Covariance

Appendix 4B: Notes on Hypothesis Test: A Qualitative Discussion

Appendix 4C: Inference and Diagnostics: A Quantitative Formulation

Appendix 4D: Notes on Continuous Entropy

Notes

Exercises and Problems

Chapter 5 - Inference in The Real World

Single Parameter Problems

Exponential Distributions and Scales

Distribution of Rainfall

The Barometric Formula

Power and Pareto Laws: Scale Free Distributions

Distribution of Gross Domestic Products

Multi-Parameter Problems

Size Distribution: An Industry Simulation

Incorporating Inequalities: Portfolio Allocation

Ecological Networks

Background

A Simple Info-Metrics Model

Efficient Network Aggregation

Summary

Notes

Exercises and Problems

Chapter 6 - Advanced Inference in The Real World

Interval Information

Theory

Conjugate Variables

Weather Pattern Analysis: The Case of New York City

Treatment Decision for Learning Disabilities

Background Information and Inferential Model

A Simulated Example

Brain Cancer: Analysis and Diagnostics

The Information

The Surprisal

Bayesian Updating: Individual Probabilities

Summary

Appendix 6A: Transformation of the Data from Intervals to Integers

Appendix 6B: Generating the Treatment Decision Data

Notes

Exercises and Problems

Chapter 7: Efficiency, Sufficiency, and Optimality

Basic Properties

Optimality

Implications of Small Variations

Efficiency

Statistical Efficiency

Computational Efficiency

Sufficiency

Concentration Theorem

Conditional Limit Theorem

Information Compression

Summary

Appendix 7A: Concentration Theorem and Chi Square

Appendix 7B: Derivation of Entropy via Stirling's Approximation

Notes

Exercises and Problems

Chapter 8 - Prior Information

A Preliminary Definition

Entropy Deficiency: Minimum Cross entropy

Grouping Property

Surprisal Analysis

Formulation

Extension: Unknown Expected Values or Dependent Variables

Transformation Groups

The Basics

Simple Examples

Maximum Entropy Priors

Empirical Priors

Priors, Treatment Effect, and Propensity Score Functions

Summary

Notes

Exercises and Problems

Chapter 9 - A Complete Info-Metrics Framework

Information, Uncertainty, and Noise

Formulation and Solution

A Simple Example with Noisy Constraints

The Concentrated Framework

A Framework for Inferring Theories and Consistent Models

Examples in an Uncertain Setting

Theory Uncertainty and Approximate Theory: Markov Process

Example: Mixed Models in Non-Ideal Setting

Uncertainty

The Optimal Solution

Lagrange Multipliers

The Stochastic Constraints

The Support Space

The Cost of Accommodating Uncertainty

Visual Representation of the Info-Metrics Framework

Adding Priors

Summary

Appendix 9A: Efficiency and Optimality

Optimality

Statistical Efficiency

Concentration Theorem

Notes

Exercises and Problems

Chapter 10 - Modeling and Theories

Core Questions

Basic Building Blocks

Problem and Entities

Information and Constraints

Incorporating Priors

Validation and Falsification

Prediction

A Detailed Social Science Example

Characterizing the Problem

Introducing the Basic Entities

Information and Constraints

Production

Consumption

Supply and Demand

Individual Preferences

Budget Constraints

The Statistical Equilibrium

Economic Entropy: Concentrated Model

Prices, Lagrange Multipliers, and Preferences

Priors, Validation, and Prediction

Model Summary

Other Classical Examples

Summary

Notes

Exercises and Problems

Chapter 11 - Causal Inference via Constraint Satisfaction

Definitions

Info-Metrics and Nonmonotonic Reasoning

Nonmonotonic Reasoning and Grouping

Typicality and Info-Metrics

The Principle of Causation

Info-Metrics and Causal Inference

Causality, Inference, and Markov Transition Probabilities: An Example

The Model

Inferred Causal Influence

Summary

Notes

Exercises and Problems

Chapter 12 - Info-Metrics and Statistical Inference: Discrete Problems

Discrete Choice Models: Statement of the Problem

Example: A Die and Discrete Choice Models

Definitions and Problem Specification

The Unconstrained Model as a Maximum Likelihood

The Constrained Optimization Model

The Info-Metrics Framework: A Generalized Likelihood

Real World Examples

Tailoring Political Messages and Testing the Impact of Negative Messages

Background on the Congressional Race and the Survey

Inference, Prediction, and the Effect of Different Messages

Is there Racial Discrimination in Home Mortgage Lending?

Background on Loans, Minorities, and Sample Size

Inference, Marginal Effects, Prediction, and Discrimination

The Benefits of Info-Metrics for Inference in Discrete Choice Problems

Summary

Notes

Exercises and Problems

Chapter 13 - Info-Metrics and Statistical Inference: Continuous Problems

Continuous Regression Models: Statement of the Problem

Definitions and Problem Specification

Unconstrained Models in Traditional Inference

Rethinking the Problem as a Constrained Optimization

A Basic Model

A General Information-Theoretic Model

Generalized Entropies

Information-Theoretic Methods of Inference: Zero-Moment Conditions

Specific Cases: Empirical and Euclidean Likelihoods

Exploring a Power Law: Shannon Entropy vs. Empirical Likelihood

Theoretical and Empirical Examples

Information-Theoretic Methods of Inference: Stochastic Moment Conditions

The Support Spaces

A Simulated Example

Misspecification

The Benefits of Info-Metrics for Inference in Continuous Problems

Information and Model Comparison

Summary

Appendix 13A: Generalized Method of Moments and Info-Metrics

Background

Definition and Traditional Formulation

The Information-Theoretic Solution

An Example in an Ideal Setting

Extension: GMM and the Info-Metrics Framework

Appendix 13B: Bayesian Method of Moments and Info-Metrics

Notes

Exercises and Problems

Chapter 14 - New Applications Across Disciplines

Option Pricing

Simple Case: One Option

Generalized Case: Inferring the Equilibrium Distribution

Implications and Significance

Predicting Coronary Artery Disease

Data and Definitions

Analyses and Results

The Complete Sample

Out of Sample Prediction

Sensitivity Analysis and Simulated Scenarios

Implications and Significance

Improved Election Prediction Using Priors on Individuals

Analyses and Results

The Data

The Priors and Analyses

Implications and Significance

Predicting Dose Effect: Drug-Induced Liver Injury

Medical Background and Objective

Data and Definitions

Inference and Predictions

A Linear Model

Analyzing the Residuals: Extreme Events

Implications and Significance

Summary

Notes

Exercises and Problems

Epilogue

Appendices

List of Symbols

References

Index