{ "cells": [ { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "********************************\n", "Common Probability Distributions\n", "********************************\n", "\n", "- When fitting probability models to data, it is necessary to know the\n", " uncertainty of the fit.\n", "\n", " - This uncertainty is represented as a probability distribution over the\n", " parameters of the fitted model.\n", " - See Table 3.1 and 3.1 for a summary.\n", "\n", "- If the posterior distributions are in the same family as the prior\n", " distributions, then the prior and the posterior are called conjugate\n", " distributions.\n", "\n", " - The prior then is known as the conjugate prior.\n", " - When a distribution is multiplied with its conjugate, the result is\n", " proportional to a new distribution which has the same form as the conjugate." ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "Exercise 3.1\n", "============\n", "\n", "Let :math:`x` be a Bernoulli distributed random variable parameterized by\n", ":math:`\\lambda`. The mean and variance are defined as\n", "\n", ".. math::\n", "\n", " \\DeclareMathOperator{\\BernDist}{Bern}\n", " \\mathrm{E}[X] &= \\sum_{x \\in X} x \\BernDist_x[\\lambda]\\\\\n", " &= 0 (1 - \\lambda) + 1 (\\lambda)\\\\\n", " &= \\lambda\n", "\n", "and\n", "\n", ".. math::\n", "\n", " \\mathrm{E}\\left[ (X - \\mathrm{E}[X])^2 \\right]\n", " &= \\mathrm{E}[X^2] - \\mathrm{E}[X]^2\n", " & \\quad & \\text{Exercise 2.10}\\\\\n", " &= \\sum_{x \\in X} x^2 \\BernDist_x[\\lambda] - \\lambda^2\n", " & \\quad & \\text{(2.12) where } f(X) \\mapsto f[x] = x^2\\\\\n", " &= 0^2 (1 - \\lambda) + 1^2 (\\lambda) - \\lambda^2\\\\\n", " &= \\lambda (1 - \\lambda).\n", "\n", "See :ref:`Exercise 2.10 ` for more details." ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ ".. _prince2012computer-ex-3.2:\n", "\n", "Exercise 3.2\n", "============\n", "\n", "A useful fact is the relationship between the beta function and the gamma\n", "function:\n", "\n", ".. math::\n", "\n", " B(x, y)\n", " &= \\int_0^1 t^{x - 1} (1 - t)^{y - 1} dt\\\\\n", " &= \\frac{\\Gamma[x] \\Gamma[y]}{\\Gamma[x + y]}\n", " & \\quad & \\text{convolution integral property}\n", "\n", "where :math:`\\Gamma[z] = \\int_0^\\infty t^{z - 1} e^{-t} dt`.\n", "\n", "Finding the mode (position of the peak) of the beta distribution is equivalent\n", "to finding the parameter that maximizes the beta distribution:\n", "\n", ".. math::\n", "\n", " \\DeclareMathOperator{\\BetaDist}{Beta}\n", " \\frac{\\partial}{\\partial \\lambda} \\BetaDist_\\lambda[\\alpha, \\beta]\n", " &= B(\\alpha, \\beta)^{-1} \\frac{\\partial}{\\partial \\lambda}\n", " \\left(\n", " \\lambda^{\\alpha - 1} (1 - \\lambda)^{\\beta - 1}\n", " \\right)\\\\\n", " 0 &= B(\\alpha, \\beta)^{-1}\n", " \\left[\n", " (\\alpha - 1) (1) \\lambda^{\\alpha - 2} (1 - \\lambda)^{\\beta - 1} +\n", " (\\beta - 1) (-1) \\lambda^{\\alpha - 1} (1 - \\lambda)^{\\beta - 2}\n", " \\right]\\\\\n", " 0 &= \\lambda^{\\alpha - 2} (1 - \\lambda)^{\\beta - 2}\n", " \\left[ (\\alpha - 1) (1 - \\lambda) - (\\beta - 1) \\lambda \\right]\\\\\n", " 0 &= -\\lambda \\left[ (\\alpha - 1) + (\\beta - 1) \\right] + (\\alpha - 1)\\\\\n", " \\lambda &= \\frac{\\alpha - 1}{\\alpha + \\beta - 2}" ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ ".. _prince2012computer-ex-3.3:\n", "\n", "Exercise 3.3\n", "============\n", "\n", "Notice that :math:`1 - \\mu = \\frac{\\beta}{\\alpha + \\beta}` and\n", ":math:`\\sigma^2 = \\frac{\\mu (1 - \\mu)}{\\alpha + \\beta + 1} \\iff\n", "\\frac{\\mu (1 - \\mu)}{\\sigma^2} = \\alpha + \\beta + 1`.\n", "\n", ".. math::\n", "\n", " \\alpha = \\mu \\left( \\frac{\\mu (1 - \\mu)}{\\sigma^2} - 1 \\right)\n", "\n", ".. math::\n", "\n", " \\beta &= \\frac{\\mu (1 - \\mu)}{\\sigma^2} - 1 - \\alpha\\\\\n", " &= \\frac{\\mu (1 - \\mu)}{\\sigma^2} - 1 -\n", " \\mu \\left( \\frac{\\mu (1 - \\mu)}{\\sigma^2} - 1 \\right)\\\\\n", " &= (1 - \\mu) \\frac{\\mu (1 - \\mu)}{\\sigma^2} + \\mu - 1\\\\\n", " &= (1 - \\mu) \\left( \\frac{\\mu (1 - \\mu)}{\\sigma^2} - 1 \\right)" ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "Exercise 3.4\n", "============\n", "\n", "Recall that :math:`\\BetaDist_\\lambda[\\alpha, \\beta] =\n", "B(\\alpha, \\beta)^{-1} \\lambda^{\\alpha - 1} (1 - \\lambda)^{\\beta - 1}`. Let\n", ":math:`x` denote :math:`\\lambda` for\n", "\n", ".. math::\n", "\n", " Pr(x \\mid \\boldsymbol{\\theta})\n", " &= a[x] \\exp\\left(\n", " \\mathbf{b}[\\boldsymbol{\\theta}]^\\top \\mathbf{c}[x] -\n", " d[\\boldsymbol{\\theta}]\n", " \\right)\\\\\n", " &= a[x]\n", " \\exp\\left( \\mathbf{b}[\\boldsymbol{\\theta}]^\\top \\mathbf{c}[x] \\right)\n", " \\exp\\left( -d[\\boldsymbol{\\theta}] \\right).\n", "\n", "The Beta distribution can be represented in the generalized form of the\n", "exponential family as follows:\n", "\n", ".. math::\n", "\n", " \\begin{gather*}\n", " a[x] = 1\\\\\n", " \\mathbf{b}[\\boldsymbol{\\theta}] =\n", " \\begin{bmatrix} \\alpha - 1\\\\ \\beta - 1 \\end{bmatrix}\\\\\n", " \\mathbf{c}[x] = \\begin{bmatrix} \\log(x)\\\\ \\log(1 - x) \\end{bmatrix}\\\\\n", " d[\\boldsymbol{\\theta}] = \\log{B(\\alpha, \\beta)}.\n", " \\end{gather*}" ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "Exercise 3.5\n", "============\n", "\n", "Given :math:`z > 0`,\n", "\n", ".. math::\n", "\n", " \\Gamma[z + 1]\n", " &= \\int_0^\\infty t^{(z + 1) - 1} e^{-t} dt\\\\\n", " &= \\left[ -t^z e^{-t} \\right]_0^\\infty -\n", " \\int_0^\\infty -e^{-t} z t^{z - 1} dt\n", " & \\quad & \\text{integration by parts with }\n", " u = t^z, dv = e^{-t} dt, v = -e^{-t}, du = z t^{z - 1} dt\\\\\n", " &= z \\Gamma[z]\n", "\n", "because\n", "\n", ".. math::\n", "\n", " \\left[ -t^z e^{-t} \\right]_0^\\infty\n", " &= \\lim_{t \\rightarrow \\infty} -t^z e^{-t} - \\left( -(0)^z e^{-0} \\right)\\\\\n", " &= \\lim_{t \\rightarrow \\infty} -\\frac{t^z}{e^t}\n", " & \\quad & \\text{convert to indeterminate form } \\frac{\\infty}{\\infty}\\\\\n", " &= \\lim_{t \\rightarrow \\infty} -\\frac{z!}{e^t}\n", " & \\quad & \\text{apply L'Hopital's rule } z + 1 \\text{ times}\\\\\n", " &= 0." ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "Exercise 3.6\n", "============\n", "\n", ".. math::\n", "\n", " \\DeclareMathOperator{\\NormDist}{Norm}\n", " Pr(x \\mid \\mu) Pr(\\mu)\n", " &= \\NormDist_x[\\mu, 1.0] \\NormDist_\\mu[\\mu_p, \\sigma_p^2]\\\\\n", " &= \\frac{1}{\\sqrt{2 \\pi}}\n", " \\exp\\left[-0.5 (x - \\mu)^2 \\right]\n", " \\frac{1}{\\sqrt{2 \\pi \\sigma_p^2}}\n", " \\exp\\left[-0.5 \\frac{(\\mu - \\mu_p)^2}{\\sigma_p^2}\\right]\\\\\n", " &= \\frac{1}{2 \\pi \\sigma}\n", " \\exp\\left[\n", " -0.5 \\left(\n", " x^2 - 2 \\mu x + \\mu^2 +\n", " \\frac{\\mu^2}{\\sigma_p^2} -\n", " \\frac{2 \\mu \\mu_p}{\\sigma_p^2} +\n", " \\frac{\\mu_p^2}{\\sigma_p^2}\n", " \\right)\n", " \\right]\\\\\n", " &= \\frac{1}{2 \\pi \\sigma}\n", " \\exp\\left[ -0.5 \\left( x^2 + \\frac{\\mu_p^2}{\\sigma_p^2} \\right) \\right]\n", " \\exp\\left[\n", " -0.5 \\left(\n", " \\mu^2 + \\frac{\\mu^2}{\\sigma_p^2} -\n", " 2 \\mu x - \\frac{2 \\mu \\mu_p}{\\sigma_p^2}\n", " \\right)\n", " \\right]\\\\\n", " &= \\kappa_1\n", " \\exp\\left[\n", " -0.5 \\left(\n", " \\frac{\\sigma_p^2 + 1}{\\sigma_p^2} \\mu^2 -\n", " 2 \\frac{\\sigma_p^2 x + \\mu_p}{\\sigma_p^2} \\mu\n", " \\right)\n", " \\right]\\\\\n", " &= \\kappa_1\n", " \\exp\\left[\n", " \\frac{-0.5 (\\sigma_p^2 + 1)}{\\sigma_p^2}\n", " \\left(\n", " \\mu^2 - 2 \\frac{\\sigma_p^2 x + \\mu_p}{\\sigma_p^2 + 1} \\mu\n", " \\right)\n", " \\right]\n", " & \\quad & \\text{set up for completing the square}\\\\\n", " &= \\kappa_1\n", " \\exp\\left[\n", " \\frac{-0.5}{\\sigma_p^2 (\\sigma_p^2 + 1)^{-1}}\n", " \\left(\n", " \\mu^2 - 2 \\frac{\\sigma_p^2 x + \\mu_p}{\\sigma_p^2 + 1} \\mu +\n", " \\left( \\frac{\\sigma_p^2 x + \\mu_p}{\\sigma_p^2 + 1} \\right)^2\n", " \\right)\n", " \\right]\n", " \\exp\\left[\n", " \\frac{0.5}{\\sigma_p^2 (\\sigma_p^2 + 1)^{-1}}\n", " \\left( \\frac{\\sigma_p^2 x + \\mu_p}{\\sigma_p^2 + 1} \\right)^2\n", " \\right]\\\\\n", " &= \\kappa_2\n", " \\exp\\left[\n", " -0.5\n", " \\frac{\n", " \\left( \\mu - \\frac{\\sigma_p^2 x + \\mu_p}{\\sigma_p^2 + 1} \\right)^2\n", " }{\n", " \\sigma_p^2 (\\sigma_p^2 + 1)^{-1}\n", " }\n", " \\right]\\\\\n", " &= \\kappa_3 \\NormDist_\\mu \\left[\n", " \\frac{\\sigma_p^2 x + \\mu_p}{\\sigma_p^2 + 1},\n", " \\sigma_p^2 (\\sigma_p^2 + 1)^{-1}\n", " \\right]" ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "Exercise 3.7\n", "============\n", "\n", "Recall that\n", "\n", ".. math::\n", "\n", " \\NormDist_x[\\mu, \\sigma^2]\n", " &= \\frac{1}{\\sqrt{2 \\pi \\sigma^2}}\n", " \\exp\\left[ -0.5 \\frac{(x - \\mu)^2}{\\sigma^2} \\right]\\\\\n", " &= \\frac{1}{\\sqrt{2 \\pi \\sigma^2}}\n", " \\exp\\left[ \\frac{-0.5}{\\sigma^2} (x^2 - 2 x \\mu + \\mu^2) \\right].\n", "\n", "The univariate normal distribution can be represented in the generalized form of\n", "the exponential family as follows:\n", "\n", ".. math::\n", "\n", " Pr(x \\mid \\boldsymbol{\\theta})\n", " &= a[x] \\exp\\left(\n", " \\mathbf{b}[\\boldsymbol{\\theta}]^\\top \\mathbf{c}[x] -\n", " d[\\boldsymbol{\\theta}]\n", " \\right)\\\\\n", " &= a[x]\n", " \\exp\\left(\n", " \\mathbf{b}[\\boldsymbol{\\theta}]^\\top \\mathbf{c}[x]\n", " \\right)\n", " \\exp\\left( -d[\\boldsymbol{\\theta}] \\right)\n", "\n", "where\n", "\n", ".. math::\n", "\n", " \\begin{gather*}\n", " a[x] = 1\\\\\n", " \\mathbf{b}[\\boldsymbol{\\theta}] =\n", " -\\frac{0.5}{\\sigma^2} \\begin{bmatrix} 1\\\\ -2 \\mu \\end{bmatrix}\\\\\n", " \\mathbf{c}[x] = \\begin{bmatrix} x^2\\\\ x \\end{bmatrix}\\\\\n", " d[\\boldsymbol{\\theta}] =\n", " 0.5 \\frac{\\mu^2}{\\sigma^2} + \\log{\\sqrt{2 \\pi \\sigma^2}}.\n", " \\end{gather*}" ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "Exercise 3.8\n", "============\n", "\n", "Finding the mode (position of the peak) of the normal scaled inverse gamma\n", "distribution is equivalent to finding its maximum.\n", "\n", ".. math::\n", "\n", " \\DeclareMathOperator{\\NormInvGamDist}{NormInvGam}\n", " \\frac{\\partial}{\\partial \\mu}\n", " \\NormInvGamDist_{\\mu, \\sigma^2}[\\alpha, \\beta, \\gamma, \\delta]\n", " &= \\frac{\\sqrt{\\gamma}}{\\sigma \\sqrt{2 \\pi}}\n", " \\frac{\\beta^\\alpha}{\\Gamma[\\alpha]}\n", " \\left( \\frac{1}{\\sigma^2} \\right)^{\\alpha + 1}\n", " \\frac{\\partial}{\\partial \\mu} \\exp\\left[\n", " -\\frac{2\\beta + \\gamma(\\delta - \\mu)^2}{2 \\sigma^2}\n", " \\right]\\\\\n", " 0 &= \\NormInvGamDist_{\\mu, \\sigma^2}[\\alpha, \\beta, \\gamma, \\delta]\n", " \\left( \\frac{\\gamma (\\delta - \\mu)}{\\sigma^2} \\right)\\\\\n", " \\mu &= \\delta\n", "\n", ".. math::\n", "\n", " \\frac{\\partial}{\\partial \\sigma}\n", " \\NormInvGamDist_{\\mu, \\sigma^2}[\\alpha, \\beta, \\gamma, \\delta]\n", " &= \\frac{\\beta^\\alpha}{\\Gamma[\\alpha]} \\sqrt{\\frac{\\gamma}{2 \\pi}}\n", " \\frac{\\partial}{\\partial \\sigma} \\left[\n", " \\sigma^{-2 \\alpha - 3}\n", " \\exp\\left[\n", " -\\frac{2\\beta + \\gamma(\\delta - \\mu)^2}{2 \\sigma^2}\n", " \\right]\n", " \\right]\\\\\n", " 0 &= (-2 \\alpha - 3) \\sigma^{-1}\n", " \\NormInvGamDist_{\\mu, \\sigma^2}[\\alpha, \\beta, \\gamma, \\delta] +\n", " \\NormInvGamDist_{\\mu, \\sigma^2}[\\alpha, \\beta, \\gamma, \\delta]\n", " \\left( \\frac{2\\beta + \\gamma(\\delta - \\mu)^2}{\\sigma^3} \\right)\\\\\n", " &= (-2 \\alpha - 3) \\sigma^2 +\n", " \\left( 2\\beta + \\gamma(\\delta - \\mu)^2 \\right)\\\\\n", " \\sigma^2 &= \\frac{2\\beta + \\gamma(\\delta - \\mu)^2}{2 \\alpha + 3}" ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ ".. _prince2012computer-ex-3.9:\n", "\n", "Exercise 3.9\n", "============\n", "\n", ".. math::\n", "\n", " \\prod_{i = 1}^I \\text{Bern}_{x_i}[\\lambda]\n", " \\cdot \\text{Beta}_\\lambda[\\alpha, \\beta]\n", " &= \\prod_{i = 1}^I \\lambda^{x_i} (1 - \\lambda)^{1 - x_i} \\cdot\n", " \\frac{\\Gamma[\\alpha + \\beta]}{\\Gamma[\\alpha] \\Gamma[\\beta]}\n", " \\lambda^{\\alpha - 1} (1 - \\lambda)^{\\beta - 1}\\\\\n", " &= \\lambda^{\\sum_i x_i} (1 - \\lambda)^{\\sum_i 1 - x_i}\n", " \\frac{\\Gamma[\\alpha + \\beta]}{\\Gamma[\\alpha] \\Gamma[\\beta]}\n", " \\lambda^{\\alpha - 1} (1 - \\lambda)^{\\beta - 1}\\\\\n", " &= \\frac{\\Gamma[\\alpha + \\beta]}{\\Gamma[\\alpha] \\Gamma[\\beta]}\n", " \\lambda^{\\alpha - 1 + \\sum_i x_i}\n", " (1 - \\lambda)^{\\beta - 1 + I - \\sum x_i}\\\\\n", " &= \\frac{\\Gamma[\\alpha + \\beta]}{\\Gamma[\\alpha] \\Gamma[\\beta]}\n", " \\frac{\n", " \\Gamma\\left[ \\alpha + \\sum_i x_i \\right]\n", " \\Gamma\\left[ \\beta + I - \\sum_i x_i \\right]\n", " }{\n", " \\Gamma[\\alpha + \\beta + I]\n", " }\n", " \\BetaDist_\\lambda\\left[\n", " \\alpha + \\sum_i x_i, \\beta + I - \\sum_i x_i\n", " \\right]" ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ ".. _prince2012computer-ex-3.10:\n", "\n", "Exercise 3.10\n", "=============\n", "\n", ".. math::\n", "\n", " \\DeclareMathOperator{\\CatDist}{Cat}\n", " \\DeclareMathOperator{\\DirDist}{Dir}\n", " \\prod_{i = 1}^I \\CatDist_{x_i}[\\lambda_{1 \\ldots K}] \\cdot\n", " \\DirDist_{\\lambda_{1 \\ldots K}}[\\alpha_{1 \\ldots K}]\n", " &= \\prod_{i = 1}^I \\prod_{j = 1}^K \\lambda_j^{x_{ij}} \\cdot\n", " \\frac{\n", " \\Gamma\\left[ \\sum_{j = 1}^K \\alpha_j \\right]\n", " }{\n", " \\prod_{j = 1}^K \\Gamma[\\alpha_j]\n", " }\n", " \\prod_{j = 1}^K \\lambda_j^{\\alpha_j - 1}\\\\\n", " &= \\frac{\n", " \\Gamma\\left[ \\sum_{j = 1}^K \\alpha_j \\right]\n", " }{\n", " \\prod_{j = 1}^K \\Gamma[\\alpha_j]\n", " }\n", " \\prod_{j = 1}^K \\lambda_j^{\\alpha_j - 1 + N_j}\n", " & \\quad & N_j = \\sum_{i = 1}^I x_{ij}\\\\\n", " &= \\frac{\n", " \\Gamma\\left[ \\sum_{j = 1}^K \\alpha_j \\right]\n", " }{\n", " \\prod_{j = 1}^K \\Gamma[\\alpha_j]\n", " }\n", " \\frac{\n", " \\prod_{j = 1}^K \\Gamma[\\alpha_j + N_j]\n", " }{\n", " \\Gamma\\left[ \\sum_{j = 1}^K \\alpha_j + N_j \\right]\n", " }\n", " \\DirDist_{\\lambda_{1 \\ldots K}}[\\alpha_1 + N_1, \\ldots, \\alpha_K + N_K]\\\\\n", " &= \\frac{\n", " \\Gamma\\left[ \\sum_{j = 1}^K \\alpha_j \\right]\n", " }{\n", " \\prod_{j = 1}^K \\Gamma[\\alpha_j]\n", " }\n", " \\frac{\n", " \\prod_{j = 1}^K \\Gamma[\\alpha_j + N_j]\n", " }{\n", " \\Gamma\\left[ I + \\sum_{j = 1}^K \\alpha_j \\right]\n", " }\n", " \\DirDist_{\\lambda_{1 \\ldots K}}[\\alpha_1 + N_1, \\ldots, \\alpha_K + N_K]\n", " & \\quad & \\sum_j N_j = I" ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ ".. _prince2012computer-ex-3.11:\n", "\n", "Exercise 3.11\n", "=============\n", "\n", ".. math::\n", "\n", " & \\prod_{i = 1}^I \\text{Norm}_{x_i}[\\mu, \\sigma^2] \\cdot\n", " \\NormInvGamDist_{\\mu, \\sigma^2}[\\alpha, \\beta, \\gamma, \\delta]\\\\\n", " &= \\prod_{i = 1}^I\n", " \\frac{1}{\\sqrt{2 \\pi \\sigma^2}}\n", " \\exp\\left[ -\\frac{(x_i - \\mu)^2}{2 \\sigma^2} \\right] \\cdot\n", " \\frac{\\sqrt{\\gamma}}{\\sigma \\sqrt{2 \\pi}}\n", " \\frac{\\beta^\\alpha}{\\Gamma[\\alpha]}\n", " \\left( \\frac{1}{\\sigma^2} \\right)^{\\alpha + 1}\n", " \\exp\\left[\n", " -\\frac{2\\beta + \\gamma(\\delta - \\mu)^2}{2 \\sigma^2}\n", " \\right]\\\\\n", " &= \\left( 2 \\pi \\sigma^2 \\right)^{-I / 2}\n", " \\frac{\\sqrt{\\gamma}}{\\sigma \\sqrt{2 \\pi}}\n", " \\frac{\\beta^\\alpha}{\\Gamma[\\alpha]}\n", " \\left( \\frac{1}{\\sigma^2} \\right)^{\\alpha + 1}\n", " \\exp\\left[\n", " -\\frac{1}{2 \\sigma^2}\n", " \\left(\n", " 2\\beta + \\gamma(\\delta - \\mu)^2 + \\sum_i (x_i - \\mu)^2\n", " \\right)\n", " \\right]\\\\\n", " &= \\frac{1}{(2 \\pi)^{I / 2}}\n", " \\frac{\\sqrt{\\gamma}}{\\sigma \\sqrt{2 \\pi}}\n", " \\frac{\\beta^\\alpha}{\\Gamma[\\alpha]}\n", " \\left( \\frac{1}{\\sigma^2} \\right)^{\\alpha + \\frac{I}{2} + 1}\n", " \\exp\\left[\n", " -\\frac{1}{2 \\sigma^2}\n", " \\left(\n", " 2\\beta + \\gamma \\delta^2 - 2 \\gamma \\delta \\mu + \\gamma \\mu^2 +\n", " \\sum_i x_i^2 - 2 \\mu \\sum_i x_i + I \\mu^2\n", " \\right)\n", " \\right]\\\\\n", " &= \\frac{1}{(2 \\pi)^{I / 2}}\n", " \\frac{\\sqrt{\\gamma}}{\\sigma \\sqrt{2 \\pi}}\n", " \\frac{\\beta^\\alpha}{\\Gamma[\\alpha]}\n", " \\left( \\frac{1}{\\sigma^2} \\right)^{\\alpha + \\frac{I}{2} + 1}\n", " \\exp\\left[\n", " -\\frac{1}{2 \\sigma^2}\n", " \\left(\n", " 2\\beta + \\sum_i x_i^2 + \\gamma \\delta^2 - 2 \\gamma \\delta \\mu -\n", " 2 \\mu \\sum_i x_i + \\tilde{\\gamma} \\mu^2\n", " \\right)\n", " \\right]\n", " & \\quad & \\text{swapped in } \\tilde{\\gamma}\\\\\n", " &= \\frac{1}{(2 \\pi)^{I / 2}}\n", " \\frac{\\sqrt{\\gamma}}{\\sigma \\sqrt{2 \\pi}}\n", " \\frac{\\beta^\\alpha}{\\Gamma[\\alpha]}\n", " \\left( \\frac{1}{\\sigma^2} \\right)^{\\alpha + \\frac{I}{2} + 1}\n", " \\exp\\left[\n", " -\\frac{1}{2 \\sigma^2}\n", " \\left(\n", " 2 \\tilde{\\beta} +\n", " \\frac{(\\gamma \\delta + \\sum_i x_i)^2}{\\tilde{\\gamma}} -\n", " 2 \\gamma \\delta \\mu - 2 \\mu \\sum_i x_i + \\tilde{\\gamma} \\mu^2\n", " \\right)\n", " \\right]\n", " & \\quad & \\text{swapped in } \\tilde{\\beta}\\\\\n", " &= \\frac{1}{(2 \\pi)^{I / 2}}\n", " \\frac{\\sqrt{\\gamma}}{\\sigma \\sqrt{2 \\pi}}\n", " \\frac{\\beta^\\alpha}{\\Gamma[\\alpha]}\n", " \\left( \\frac{1}{\\sigma^2} \\right)^{\\tilde{\\alpha} + 1}\n", " \\exp\\left[\n", " -\\frac{2 \\tilde{\\beta} + \\tilde{\\gamma} (\\tilde{\\delta} - \\mu)^2}\n", " {2 \\sigma^2}\n", " \\right]\n", " & \\quad & \\text{swapped in } \\tilde{\\alpha} \\text{ and } \\tilde{\\delta}\\\\\n", " &= \\kappa \\NormInvGamDist_{\\mu, \\sigma^2}\\left[\n", " \\tilde{\\alpha}, \\tilde{\\beta}, \\tilde{\\gamma}, \\tilde{\\delta}\n", " \\right]\n", " & \\quad & \\text{swapped in } \\kappa" ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "Exercise 3.12\n", "=============\n", "\n", "A useful trace property is\n", "\n", ".. math::\n", "\n", " \\DeclareMathOperator{\\tr}{\\mathrm{tr}}\n", " \\tr\\left[ z z^\\top A^{-1} \\right] = z^\\top A^{-1} z.\n", "\n", ".. math::\n", "\n", " \\DeclareMathOperator{\\NorIWisDist}{NorIWis}\n", " & \\prod_{i = 1}^I \\NormDist_{x_i}[\\boldsymbol{\\mu}, \\boldsymbol{\\Sigma}]\n", " \\cdot\n", " \\NorIWisDist_{\\boldsymbol{\\mu}, \\boldsymbol{\\Sigma}}[\n", " \\alpha, \\boldsymbol{\\Psi}, \\gamma, \\boldsymbol{\\delta}\n", " ]\\\\\n", " &= \\prod_{i = 1}^I\n", " \\frac{1}{\n", " (2 \\pi)^{D / 2} \\left\\vert \\boldsymbol{\\Sigma} \\right\\vert^{1/2}\n", " }\n", " \\exp\\left[\n", " -0.5 (\\mathbf{x}_i - \\boldsymbol{\\mu})^\\top \\boldsymbol{\\Sigma}^{-1}\n", " (\\mathbf{x}_i - \\boldsymbol{\\mu})\n", " \\right]\n", " \\cdot\n", " \\frac{\n", " \\gamma^{D / 2} \\left\\vert \\boldsymbol{\\Psi} \\right\\vert^{\\alpha / 2}\n", " \\exp\\left[\n", " -0.5 \\left(\n", " \\tr\\left[ \\boldsymbol{\\Psi} \\boldsymbol{\\Sigma}^{-1} \\right] +\n", " \\gamma (\\boldsymbol{\\mu} - \\boldsymbol{\\delta})^\\top\n", " \\boldsymbol{\\Sigma}^{-1}\n", " (\\boldsymbol{\\mu} - \\boldsymbol{\\delta})\n", " \\right)\n", " \\right]\n", " }{\n", " 2^{\\alpha D / 2} (2 \\pi)^{D / 2}\n", " \\left\\vert \\boldsymbol{\\Sigma} \\right\\vert^{(\\alpha + D + 2) / 2}\n", " \\Gamma_D[\\alpha / 2]\n", " }\n", "\n", ".. math::\n", "\n", " &= \\frac{\n", " \\left\\vert \\boldsymbol{\\Psi} \\right\\vert^{\\alpha / 2} \\gamma^{D / 2}\n", " }{\n", " \\pi^{ID / 2} \\Gamma_D[\\alpha / 2]\n", " }\n", " \\frac{\n", " \\exp\\left[\n", " -0.5 \\left(\n", " \\tr\\left[ \\boldsymbol{\\Psi} \\boldsymbol{\\Sigma}^{-1} \\right] +\n", " \\gamma (\\boldsymbol{\\mu} - \\boldsymbol{\\delta})^\\top\n", " \\boldsymbol{\\Sigma}^{-1}\n", " (\\boldsymbol{\\mu} - \\boldsymbol{\\delta}) +\n", " \\sum_i\n", " (\\mathbf{x}_i - \\boldsymbol{\\mu})^\\top\n", " \\boldsymbol{\\Sigma}^{-1}\n", " (\\mathbf{x}_i - \\boldsymbol{\\mu})\n", " \\right)\n", " \\right]\n", " }{\n", " 2^{\\tilde{\\alpha} D / 2} (2 \\pi)^{D / 2}\n", " \\left\\vert\n", " \\boldsymbol{\\Sigma}\n", " \\right\\vert^{(\\tilde{\\alpha} + D + 2) / 2}\n", " }\n", " & \\quad & \\text{swapped in } \\tilde{\\alpha}\\\\\n", " &= \\frac{\n", " \\left\\vert \\boldsymbol{\\Psi} \\right\\vert^{\\alpha / 2} \\gamma^{D / 2}\n", " }{\n", " \\pi^{ID / 2} \\Gamma_D[\\alpha / 2]\n", " }\n", " \\frac{\n", " \\exp\\left[\n", " -0.5 \\left(\n", " \\tr\\left[ \\boldsymbol{\\Psi} \\boldsymbol{\\Sigma}^{-1} \\right] +\n", " \\gamma \\boldsymbol{\\mu}^\\top \\boldsymbol{\\Sigma}^{-1}\n", " \\boldsymbol{\\mu} -\n", " 2 \\gamma \\boldsymbol{\\mu}^\\top \\boldsymbol{\\Sigma}^{-1}\n", " \\boldsymbol{\\delta} +\n", " \\gamma \\boldsymbol{\\delta}^\\top \\boldsymbol{\\Sigma}^{-1}\n", " \\boldsymbol{\\delta} +\n", " \\sum_i\n", " \\mathbf{x}_i^\\top \\boldsymbol{\\Sigma}^{-1} \\mathbf{x}_i -\n", " 2 \\sum_i\n", " \\boldsymbol{\\mu}^\\top \\boldsymbol{\\Sigma}^{-1} \\mathbf{x}_i +\n", " I \\boldsymbol{\\mu}^\\top \\boldsymbol{\\Sigma}^{-1} \\boldsymbol{\\mu}\n", " \\right)\n", " \\right]\n", " }{\n", " 2^{\\tilde{\\alpha} D / 2} (2 \\pi)^{D / 2}\n", " \\left\\vert\n", " \\boldsymbol{\\Sigma}\n", " \\right\\vert^{(\\tilde{\\alpha} + D + 2) / 2}\n", " }\\\\\n", " &= \\frac{\n", " \\left\\vert \\boldsymbol{\\Psi} \\right\\vert^{\\alpha / 2} \\gamma^{D / 2}\n", " }{\n", " \\pi^{ID / 2} \\Gamma_D[\\alpha / 2]\n", " }\n", " \\frac{\n", " \\exp\\left[\n", " -0.5 \\left(\n", " \\tr\\left[\n", " \\left(\n", " \\boldsymbol{\\Psi} +\n", " \\gamma \\boldsymbol{\\delta} \\boldsymbol{\\delta}^\\top +\n", " \\sum_i \\mathbf{x}_i \\mathbf{x}_i^\\top\n", " \\right)\n", " \\boldsymbol{\\Sigma}^{-1}\n", " \\right] +\n", " \\tilde{\\gamma} \\boldsymbol{\\mu}^\\top \\boldsymbol{\\Sigma}^{-1}\n", " \\boldsymbol{\\mu} -\n", " 2 \\gamma \\boldsymbol{\\mu}^\\top \\boldsymbol{\\Sigma}^{-1}\n", " \\boldsymbol{\\delta} -\n", " 2 \\sum_i \\boldsymbol{\\mu}^\\top \\boldsymbol{\\Sigma}^{-1}\n", " \\mathbf{x}_i\n", " \\right)\n", " \\right]\n", " }{\n", " 2^{\\tilde{\\alpha} D / 2} (2 \\pi)^{D / 2}\n", " \\left\\vert\n", " \\boldsymbol{\\Sigma}\n", " \\right\\vert^{(\\tilde{\\alpha} + D + 2) / 2}\n", " }\n", " & \\quad & \\text{swapped in } \\tilde{\\gamma}\\\\\n", " &= \\frac{\n", " \\left\\vert \\boldsymbol{\\Psi} \\right\\vert^{\\alpha / 2} \\gamma^{D / 2}\n", " }{\n", " \\pi^{ID / 2} \\Gamma_D[\\alpha / 2]\n", " }\n", " \\frac{\n", " \\exp\\left[\n", " -0.5 \\left(\n", " \\tr\\left[\n", " \\left(\n", " \\boldsymbol{\\Psi} +\n", " \\gamma \\boldsymbol{\\delta} \\boldsymbol{\\delta}^\\top +\n", " \\sum_i \\mathbf{x}_i \\mathbf{x}_i^\\top\n", " \\right)\n", " \\boldsymbol{\\Sigma}^{-1}\n", " \\right] +\n", " \\tilde{\\gamma}\n", " (\\boldsymbol{\\mu} - \\tilde{\\boldsymbol{\\delta}})^\\top\n", " \\boldsymbol{\\Sigma}^{-1}\n", " (\\boldsymbol{\\mu} - \\tilde{\\boldsymbol{\\delta}}) -\n", " \\tilde{\\gamma} \\tilde{\\boldsymbol{\\delta}}^\\top\n", " \\boldsymbol{\\Sigma}^{-1} \\tilde{\\boldsymbol{\\delta}}\n", " \\right)\n", " \\right]\n", " }{\n", " 2^{\\tilde{\\alpha} D / 2} (2 \\pi)^{D / 2}\n", " \\left\\vert\n", " \\boldsymbol{\\Sigma}\n", " \\right\\vert^{(\\tilde{\\alpha} + D + 2) / 2}\n", " }\n", " & \\quad & \\text{swapped in } \\tilde{\\boldsymbol{\\delta}}\\\\\n", " &= \\frac{\n", " \\left\\vert \\boldsymbol{\\Psi} \\right\\vert^{\\alpha / 2} \\gamma^{D / 2}\n", " }{\n", " \\pi^{ID / 2} \\Gamma_D[\\alpha / 2]\n", " }\n", " \\frac{\n", " \\exp\\left[\n", " -0.5 \\left(\n", " \\tr\\left[\n", " \\tilde{\\boldsymbol{\\Psi}} \\boldsymbol{\\Sigma}^{-1}\n", " \\right] +\n", " \\tilde{\\gamma}\n", " (\\boldsymbol{\\mu} - \\tilde{\\boldsymbol{\\delta}})^\\top\n", " \\boldsymbol{\\Sigma}^{-1}\n", " (\\boldsymbol{\\mu} - \\tilde{\\boldsymbol{\\delta}})\n", " \\right)\n", " \\right]\n", " }{\n", " 2^{\\tilde{\\alpha} D / 2} (2 \\pi)^{D / 2}\n", " \\left\\vert\n", " \\boldsymbol{\\Sigma}\n", " \\right\\vert^{(\\tilde{\\alpha} + D + 2) / 2}\n", " }\n", " & \\quad & \\text{swapped in } \\tilde{\\boldsymbol{\\Psi}}\\\\\n", " &= \\kappa \\NorIWisDist_{\\boldsymbol{\\mu}, \\boldsymbol{\\Sigma}}\\left[\n", " \\tilde{\\alpha},\n", " \\tilde{\\boldsymbol{\\Psi}},\n", " \\tilde{\\gamma},\n", " \\tilde{\\boldsymbol{\\delta}}\n", " \\right]\n", " & \\quad & \\text{swapped in } \\kappa" ] } ], "metadata": { "anaconda-cloud": {}, "celltoolbar": "Raw Cell Format", "kernelspec": { "display_name": "Python [default]", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "nbformat_minor": 0 }