{ "cells": [ { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "***********************\n", "Models for Visual Words\n", "***********************\n", "\n", "Latent Dirichlet allocation is a density model for the data in a corpus. It\n", "does not involve a \"world\" term that we wish to infer." ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "Exercise 20.1\n", "=============\n", "\n", "Section 9.9 and 9.10 describe discriminative approaches that handle multi-class\n", "problems. The data is then defined as normalized word frequencies\n", ":math:`\\mathbf{x} = [T_1, T_2, \\ldots, T_K] / \\sum_k T_k`." ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "Exercise 20.2\n", "=============\n", "\n", "The following description is the generative process for the plate notation in\n", "Figure 20.6.\n", "\n", "For each image indexed by :math:`i \\in \\{1, \\ldots, I\\}` in a corpus:\n", "\n", "1. Choose :math:`\\DeclareMathOperator{\\PoissonDist}\n", " J_i \\sim \\PoissonDist(\\xi)`.\n", "#. Choose a :math:`M`-dimensional part probability vector\n", " :math:`\\boldsymbol{\\pi}_i` from the distribution\n", " :math:`\\DeclareMathOperator{\\DirDist}{Dir}\n", " Pr(\\boldsymbol{\\pi}_i) = \\DirDist_{\\boldsymbol{\\pi}_i}[\\boldsymbol{\\alpha}]`.\n", "#. For each visual word indexed by :math:`j \\in \\{1, \\ldots, J_i\\}` in image\n", " :math:`i`:\n", "\n", " A. Choose a part :math:`p_{ij} \\in \\{1, \\ldots, M\\}` from the distribution\n", " :math:`\\DeclareMathOperator{\\CatDist}{Cat}\n", " Pr(p_{ij}) = \\CatDist_{p_{ij}}[\\boldsymbol{\\pi}_i]`.\n", " #. Choose a visual word :math:`f_{ij} \\in \\{1, \\ldots, K\\}` from the\n", " distribution\n", " :math:`Pr(f_{ij} \\mid p_{ij}) =\n", " \\CatDist_{f_{ij}}[\\boldsymbol{\\lambda}_{p_{ij}}]` where\n", " :math:`Pr(\\boldsymbol{\\lambda}_m) =\n", " \\DirDist_{\\boldsymbol{\\lambda}_m}[\\boldsymbol{\\beta}]`.\n", "\n", "The joint distribution for this generative process' graphical model is\n", "\n", ".. math::\n", "\n", " Pr(\\boldsymbol{\\pi}, \\boldsymbol{\\lambda}, \\mathbf{p}, \\mathbf{f} \\mid\n", " \\boldsymbol{\\alpha}, \\boldsymbol{\\beta})\n", " &= Pr(\\boldsymbol{\\lambda} \\mid \\boldsymbol{\\beta})\n", " Pr(\\boldsymbol{\\pi} \\mid \\boldsymbol{\\alpha})\n", " Pr(\\mathbf{p} \\mid \\boldsymbol{\\pi})\n", " Pr(\\mathbf{f} \\mid \\mathbf{p}, \\boldsymbol{\\lambda})\n", " & \\quad & \\text{(10.3)}\\\\\n", " &= \\prod_{m = 1}^M Pr(\\boldsymbol{\\lambda}_m \\mid \\boldsymbol{\\beta})\n", " \\cdot\n", " \\prod_{i = 1}^I Pr(\\boldsymbol{\\pi}_i \\mid \\boldsymbol{\\alpha})\n", " \\cdot\n", " \\prod_{i = 1}^I \\prod_{j = 1}^{J_i} Pr(p_{ij} \\mid \\boldsymbol{\\pi}_i)\n", " \\cdot\n", " \\prod_{i = 1}^I \\prod_{j = 1}^{J_i}\n", " Pr(f_{ij} \\mid \\boldsymbol{\\lambda}_{p_{ij}})\n", " & \\quad & \\text{d-Separation from generative process}\\\\\n", " &= \\prod_{m = 1}^M Pr(\\boldsymbol{\\lambda}_m \\mid \\boldsymbol{\\beta}) \\cdot\n", " \\prod_{i = 1}^I Pr(\\boldsymbol{\\pi}_i \\mid \\boldsymbol{\\alpha})\n", " \\prod_{j = 1}^{J_i}\n", " Pr(p_{ij} \\mid \\boldsymbol{\\pi}_i)\n", " Pr(f_{ij} \\mid \\boldsymbol{\\lambda}_{p_{ij}})\n", "\n", "where :math:`\\mathbf{p} = \\{ p_{ij} \\}_{i = 1, j = 1}^{I, J_i}` and\n", ":math:`\\mathbf{f} = \\{ f_{ij} \\}_{i = 1, j = 1}^{I, J_i}` are the hidden part\n", "labels and observed visual word labels respectively. This joint distribution is\n", "also known as the complete-data likelihood of an image.\n", "\n", "When the part and visual word labels are known, the Bayesian approach (or\n", "another technique like ML and MAP) can be applied to fit the parameters\n", ":math:`\\boldsymbol{\\theta} =\n", "\\left\\{\n", "\\{ \\boldsymbol{\\pi}_i \\}_{i = 1}^I,\n", "\\{ \\boldsymbol{\\lambda}_m \\}_{m = 1}^M\n", "\\right\\}` to the data. The Bayesian approach requires calculating a posterior\n", "over the parameters as\n", "\n", ".. math::\n", "\n", " Pr(\\boldsymbol{\\theta} \\mid\n", " \\mathbf{p}, \\mathbf{f}, \\boldsymbol{\\alpha}, \\boldsymbol{\\beta})\n", " &= \\frac{\n", " Pr(\\boldsymbol{\\pi}, \\boldsymbol{\\lambda}, \\mathbf{p}, \\mathbf{f} \\mid\n", " \\boldsymbol{\\alpha}, \\boldsymbol{\\beta})\n", " }{\n", " Pr(\\mathbf{p}, \\mathbf{f} \\mid \\boldsymbol{\\alpha}, \\boldsymbol{\\beta})\n", " }\\\\\n", " &= \\frac{\n", " Pr(\\boldsymbol{\\lambda} \\mid \\boldsymbol{\\beta})\n", " Pr(\\boldsymbol{\\pi} \\mid \\boldsymbol{\\alpha})\n", " Pr(\\mathbf{p} \\mid \\boldsymbol{\\pi})\n", " Pr(\\mathbf{f} \\mid \\mathbf{p}, \\boldsymbol{\\lambda})\n", " }{\n", " Pr(\\mathbf{f} \\mid \\mathbf{p}, \\boldsymbol{\\alpha}, \\boldsymbol{\\beta})\n", " Pr(\\mathbf{p} \\mid \\boldsymbol{\\alpha}, \\boldsymbol{\\beta})\n", " }\\\\\n", " &= \\frac{\n", " Pr(\\boldsymbol{\\lambda} \\mid \\boldsymbol{\\beta})\n", " Pr(\\mathbf{f} \\mid \\mathbf{p}, \\boldsymbol{\\lambda})\n", " Pr(\\boldsymbol{\\pi} \\mid \\boldsymbol{\\alpha})\n", " Pr(\\mathbf{p} \\mid \\boldsymbol{\\pi})\n", " }{\n", " \\left(\n", " \\int Pr(\\boldsymbol{\\lambda}, \\mathbf{f} \\mid\n", " \\mathbf{p}, \\boldsymbol{\\alpha}, \\boldsymbol{\\beta})\n", " d\\boldsymbol{\\lambda}\n", " \\right)\n", " \\left(\n", " \\prod_{i = 1}^I\n", " \\int Pr(\\boldsymbol{\\pi}_i, \\mathbf{p}_{i \\cdot} \\mid\n", " \\boldsymbol{\\alpha}, \\boldsymbol{\\beta})\n", " d\\boldsymbol{\\pi}_i\n", " \\right)\n", " }\n", " & \\quad & \\text{marginalization and independence from generative process}\\\\\n", " &= \\frac{\n", " \\prod_{m = 1}^M Pr(\\boldsymbol{\\lambda}_m \\mid \\boldsymbol{\\beta})\n", " \\cdot\n", " \\prod_{i = 1}^I \\prod_{j = 1}^{J_i}\n", " Pr(f_{ij} \\mid \\boldsymbol{\\lambda}_{p_{ij}}) \\cdot\n", " \\prod_{i = 1}^I Pr(\\boldsymbol{\\pi}_i \\mid \\boldsymbol{\\alpha})\n", " \\prod_{j = 1}^{J_i} Pr(p_{ij} \\mid \\boldsymbol{\\pi}_i)\n", " }{\n", " \\left(\n", " \\int\n", " \\prod_{m = 1}^M Pr(\\boldsymbol{\\lambda}_m \\mid \\boldsymbol{\\beta})\n", " \\cdot\n", " \\prod_{i = 1}^I \\prod_{j = 1}^{J_i}\n", " Pr(f_{ij} \\mid \\boldsymbol{\\lambda}_{p_{ij}})\n", " d\\boldsymbol{\\lambda}\n", " \\right)\n", " \\left(\n", " \\prod_{i = 1}^I\n", " \\int Pr(\\boldsymbol{\\pi}_i \\mid \\boldsymbol{\\alpha})\n", " \\prod_{j = 1}^{J_i}\n", " Pr(\\mathbf{p}_{i \\cdot} \\mid \\boldsymbol{\\pi}_i)\n", " d\\boldsymbol{\\pi}_i\n", " \\right)\n", " }\n", " & \\quad & \\text{d-Separation from generative process}\\\\\n", " &= \\prod_{m = 1}^M\n", " \\DirDist_{\\boldsymbol{\\lambda}_m}\\left[\n", " \\beta + F_{m1}, \\ldots, \\beta + F_{mK}\n", " \\right]\n", " \\cdot\n", " \\prod_{i = 1}^I\n", " \\DirDist_{\\boldsymbol{\\pi}_i}\\left[\n", " \\alpha + P_{i1}, \\ldots, \\alpha + P_{iM}\n", " \\right]\n", " & \\quad & \\text{Exercise 20.3 (a) and (b)}\n", "\n", "See :ref:`Exercise 20.3 ` for more details.\n", "\n", "The predictive density (probability that a new data point\n", ":math:`\\{p^*, f^*\\}` belongs to the same model) is\n", "\n", ".. math::\n", "\n", " & Pr(p^*, f^* \\mid\n", " \\mathbf{p}, \\mathbf{f}, \\boldsymbol{\\alpha}, \\boldsymbol{\\beta})\\\\\n", " &= \\int Pr(p^*, f^* \\mid \\boldsymbol{\\theta})\n", " Pr(\\boldsymbol{\\theta} \\mid\n", " \\mathbf{p}, \\mathbf{f}, \\boldsymbol{\\alpha}, \\boldsymbol{\\beta})\n", " d\\boldsymbol{\\theta}\\\\\n", " &= \\int Pr(f^* \\mid \\boldsymbol{\\lambda}_{p^*})\n", " Pr(p^* \\mid \\boldsymbol{\\pi}_*)\n", " Pr(\\boldsymbol{\\theta} \\mid\n", " \\mathbf{p}, \\mathbf{f}, \\boldsymbol{\\alpha}, \\boldsymbol{\\beta})\n", " d\\boldsymbol{\\theta}\n", " & \\quad & \\text{d-Separation from generative process}\\\\\n", " &= \\int \\CatDist_{f^*}[\\boldsymbol{\\lambda}_{p*}]\n", " \\CatDist_{p^*}[\\boldsymbol{\\pi}_*]\n", " \\cdot\n", " \\prod_{m = 1}^M\n", " \\DirDist_{\\boldsymbol{\\lambda}_m}\\left[\n", " \\beta + F_{m1}, \\ldots, \\beta + F_{mK}\n", " \\right]\n", " \\cdot\n", " \\prod_{i = 1}^I\n", " \\DirDist_{\\boldsymbol{\\pi}_i}\\left[\n", " \\alpha + P_{i1}, \\ldots, \\alpha + P_{iM}\n", " \\right]\n", " d\\boldsymbol{\\lambda} d\\boldsymbol{\\pi}\n", " & \\quad & \\text{(20.5)}\\\\\n", " &= \\frac{\\beta + F_{mk}}{K \\beta + \\sum_{k = 1}^K F_{mk}}\n", " \\frac{\\alpha + P_{im}}{M \\alpha + \\sum_{m = 1}^M P_{im}}\n", " & \\quad & \\text{(a), (b), and }\n", " \\Gamma[1 + n] = n \\Gamma[n] \\text{ for } n > 0.\n", "\n", ":cite:`tu2014dirichlet,griffiths2002gibbs,gregor2005parameter` are helpful\n", "resources in proving these relations.\n", "\n", "(a)\n", "---\n", "\n", ".. math::\n", "\n", " & \\prod_{m = 1}^M\n", " \\DirDist_{\\boldsymbol{\\lambda}_m}\\left[\n", " \\beta + F_{m1}, \\ldots, \\beta + F_{mK}\n", " \\right]\n", " \\cdot\n", " \\CatDist_{f^*}[\\boldsymbol{\\lambda}_{p^*}]\\\\\n", " &= \\left(\n", " \\prod_{m = 1}^M\n", " \\frac{\n", " \\Gamma\\left[ K \\beta + \\sum_{k = 1}^K F_{mk} \\right]\n", " }{\n", " \\prod_{k = 1}^K \\Gamma\\left[ \\beta + F_{mk} \\right]\n", " }\n", " \\prod_{k = 1}^K \\boldsymbol{\\lambda}_{mk}^{\\beta - 1 + F_{mk}}\n", " \\right)\n", " \\cdot\n", " \\prod_{k = 1}^K \\boldsymbol{\\lambda}_{p^* k}^{\\delta[f^* - k]}\n", " & \\quad & \\text{(3.9), (3.8)}\\\\\n", " &= \\left(\n", " \\prod_{m = 1}^M\n", " \\frac{\n", " \\Gamma\\left[ K \\beta + \\sum_{k = 1}^K F_{mk} \\right]\n", " }{\n", " \\prod_{k = 1}^K \\Gamma\\left[ \\beta + F_{mk} \\right]\n", " }\n", " \\prod_{k = 1}^K \\boldsymbol{\\lambda}_{mk}^{\\beta - 1 + F_{mk}}\n", " \\right)\n", " \\cdot\n", " \\prod_{m = 1}^M\n", " \\prod_{k = 1}^K \\boldsymbol{\\lambda}_{mk}^{\\tilde{F}_{mk}}\n", " & \\quad & \\tilde{F}_{mk} =\n", " \\delta\\left[ f^* - k \\right] \\delta\\left[ p^* - m \\right]\\\\\n", " &= \\prod_{m = 1}^M\n", " \\frac{\n", " \\Gamma\\left[ K \\beta + \\sum_{k = 1}^K F_{mk} \\right]\n", " }{\n", " \\prod_{k = 1}^K \\Gamma\\left[ \\beta + F_{mk} \\right]\n", " }\n", " \\prod_{k = 1}^K\n", " \\boldsymbol{\\lambda}_{mk}^{\\beta - 1 + F_{mk} + \\tilde{F}_{mk}}\\\\\n", " &= \\prod_{m = 1}^M\n", " \\frac{\n", " \\Gamma\\left[ K \\beta + \\sum_{k = 1}^K F_{mk} \\right]\n", " }{\n", " \\prod_{k = 1}^K \\Gamma\\left[ \\beta + F_{mk} \\right]\n", " }\n", " \\frac{\n", " \\prod_{k = 1}^K \\Gamma\\left[ \\beta + F_{mk} + \\tilde{F}_{mk} \\right]\n", " }{\n", " \\Gamma\\left[ K \\beta + \\sum_{k = 1}^K F_{mk} + \\tilde{F}_{mk} \\right]\n", " }\n", " \\DirDist_{\\boldsymbol{\\lambda}_m}\\left[\n", " \\beta + F_{m1} + \\tilde{F}_{m1},\n", " \\ldots,\n", " \\beta + F_{mK} + \\tilde{F}_{mK}\n", " \\right]\n", "\n", "(b)\n", "---\n", "\n", ".. math::\n", "\n", " & \\prod_{i = 1}^I \\DirDist_{\\boldsymbol{\\pi}_i}\\left[\n", " \\alpha + P_{i1}, \\ldots, \\alpha + P_{iM}\n", " \\right]\n", " \\cdot\n", " \\CatDist_{p^*}[\\boldsymbol{\\pi}_*]\\\\\n", " &= \\left(\n", " \\prod_{i = 1}^I\n", " \\frac{\n", " \\Gamma\\left[ M \\alpha + \\sum_{m = 1}^M P_{im} \\right]\n", " }{\n", " \\prod_{m = 1}^M \\Gamma\\left[ \\alpha + P_{im} \\right]\n", " }\n", " \\prod_{m = 1}^M \\boldsymbol{\\pi}_{im}^{\\alpha - 1 + P_{im}}\n", " \\right)\n", " \\cdot\n", " \\prod_{m = 1}^M \\boldsymbol{\\pi}_{*m}^{\\delta\\left[ p^* - m \\right]}\n", " & \\quad & \\text{(3.9), (3.8)}\\\\\n", " &= \\left(\n", " \\prod_{i = 1}^I\n", " \\frac{\n", " \\Gamma\\left[ M \\alpha + \\sum_{m = 1}^M P_{im} \\right]\n", " }{\n", " \\prod_{m = 1}^M \\Gamma\\left[ \\alpha + P_{im} \\right]\n", " }\n", " \\prod_{m = 1}^M \\boldsymbol{\\pi}_{im}^{\\alpha - 1 + P_{im}}\n", " \\right)\n", " \\cdot\n", " \\prod_{i = 1}^I \\prod_{m = 1}^M \\boldsymbol{\\pi}_{im}^{\\tilde{P}_{im}}\n", " & \\quad & \\tilde{P}_{im} =\n", " \\delta\\left[ p^* - m \\right] \\delta\\left[ * - i \\right]\\\\\n", " &= \\prod_{i = 1}^I\n", " \\frac{\n", " \\Gamma\\left[ M \\alpha + \\sum_{m = 1}^M P_{im} \\right]\n", " }{\n", " \\prod_{m = 1}^M \\Gamma\\left[ \\alpha + P_{im} \\right]\n", " }\n", " \\prod_{m = 1}^M\n", " \\boldsymbol{\\pi}_{im}^{\\alpha - 1 + P_{im} + \\tilde{P}_{im}}\\\\\n", " &= \\prod_{i = 1}^I\n", " \\frac{\n", " \\Gamma\\left[ M \\alpha + \\sum_{m = 1}^M P_{im} \\right]\n", " }{\n", " \\prod_{m = 1}^M \\Gamma\\left[ \\alpha + P_{im} \\right]\n", " }\n", " \\frac{\n", " \\prod_{m = 1}^M \\Gamma\\left[ \\alpha + P_{im} + \\tilde{P}_{im} \\right]\n", " }{\n", " \\Gamma\\left[\n", " M \\alpha + \\sum_{m = 1}^M P_{im} + \\tilde{P}_{im}\n", " \\right]\n", " }\n", " \\DirDist_{\\boldsymbol{\\pi}_i}\\left[\n", " \\alpha + P_{i1} + \\tilde{P}_{i1},\n", " \\ldots,\n", " \\alpha + P_{iM} + \\tilde{P}_{iM}\n", " \\right]" ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ ".. _prince2012computer-ex-20.3:\n", "\n", "Exercise 20.3\n", "=============\n", "\n", "The following is based on the derivations in\n", ":ref:`Exercise 3.10 `.\n", "\n", "The likelihood term in LDA is given by\n", "\n", ".. math::\n", "\n", " & \\int \\prod_{m = 1}^M Pr(\\boldsymbol{\\lambda}_m \\mid \\boldsymbol{\\beta})\n", " \\cdot\n", " \\prod_{i = 1}^I\n", " \\prod_{j = 1}^{J_i} Pr(f_{ij} \\mid \\boldsymbol{\\lambda}_{p_{ij}})\n", " d\\boldsymbol{\\lambda}_{1 \\ldots M}\n", " & \\quad & \\text{(20.10)}\\\\\n", " &= \\left( \\frac{\\Gamma\\left[K \\beta\\right]}{\\Gamma[\\beta]^K} \\right)^M\n", " \\prod_{m = 1}^M\n", " \\frac{\n", " \\prod_{k = 1}^K \\Gamma\\left[ \\beta + F_{mk} \\right]\n", " }{\n", " \\Gamma\\left[ K \\beta + \\sum_{k = 1}^K F_{mk} \\right]\n", " }\n", " \\cdot\n", " \\int \\prod_{m = 1}^M\n", " \\DirDist_{\\boldsymbol{\\lambda}_m}\\left[\n", " \\beta + F_{m1}, \\ldots, \\beta + F_{mK}\n", " \\right]\n", " d\\boldsymbol{\\lambda}_{1 \\ldots M}\n", " & \\quad & \\text{(a)}\\\\\n", " &= \\left( \\frac{\\Gamma\\left[K \\beta\\right]}{\\Gamma[\\beta]^K} \\right)^M\n", " \\prod_{m = 1}^M\n", " \\frac{\n", " \\prod_{k = 1}^K \\Gamma[\\beta + F_{mk}]\n", " }{\n", " \\Gamma\\left[ K \\beta + \\sum_{k = 1}^K F_{mk} \\right]\n", " }.\n", "\n", "The prior term in LDA is given by\n", "\n", ".. math::\n", "\n", " & \\prod_{i = 1}^I \\int Pr(\\boldsymbol{\\pi}_i \\mid \\boldsymbol{\\alpha})\n", " \\prod_{j = 1}^{J_i}\n", " Pr(p_{ij} \\mid \\boldsymbol{\\pi}_i) d\\boldsymbol{\\pi}_{i}\n", " & \\quad & \\text{(20.11)}\\\\\n", " &= \\prod_{i = 1}^I\n", " \\frac{\\Gamma[M \\alpha]}{\\Gamma[\\alpha]^M}\n", " \\frac{\n", " \\prod_{m = 1}^M \\Gamma\\left[ \\alpha + P_{im} \\right]\n", " }{\n", " \\Gamma\\left[ M \\alpha + \\sum_{m = 1}^M P_{im} \\right]\n", " }\n", " \\int \\DirDist_{\\boldsymbol{\\pi}_i}\\left[\n", " \\alpha + P_{i1}, \\ldots, \\alpha + P_{iM}\n", " \\right]\n", " d\\boldsymbol{\\pi}_i\n", " & \\quad & \\text{(b)}\\\\\n", " &= \\left( \\frac{\\Gamma[M \\alpha]}{\\Gamma[\\alpha]^M} \\right)^I\n", " \\prod_{i = 1}^I\n", " \\frac{\n", " \\prod_{m = 1}^M \\Gamma\\left[ \\alpha + P_{im} \\right]\n", " }{\n", " \\Gamma\\left[ M \\alpha + \\sum_{m = 1}^M P_{im} \\right]\n", " }.\n", "\n", "(a)\n", "---\n", "\n", ".. math::\n", "\n", " \\prod_{m = 1}^M Pr(\\boldsymbol{\\lambda}_m \\mid \\boldsymbol{\\beta})\n", " \\cdot\n", " \\prod_{i = 1}^I \\prod_{j = 1}^{J_i}\n", " Pr(f_{ij} \\mid \\boldsymbol{\\lambda}_{p_{ij}})\n", " &= \\prod_{m = 1}^M \\DirDist_{\\boldsymbol{\\lambda}_m}[\\boldsymbol{\\beta}]\n", " \\cdot\n", " \\prod_{i = 1}^I \\prod_{j = 1}^{J_i}\n", " \\CatDist_{f_{ij}}[\\boldsymbol{\\lambda}_{p_{ij}}]\n", " & \\quad & \\text{(20.5), (20.7)}\\\\\n", " &= \\left(\n", " \\prod_{m = 1}^M\n", " \\frac{\n", " \\Gamma\\left[ \\sum_{k = 1}^K \\boldsymbol{\\beta}_k \\right]\n", " }{\n", " \\prod_{k = 1}^K \\Gamma\\left[ \\boldsymbol{\\beta}_k \\right]\n", " }\n", " \\prod_{k = 1}^K \\boldsymbol{\\lambda}_{mk}^{\\boldsymbol{\\beta}_k - 1}\n", " \\right)\n", " \\cdot\n", " \\prod_{i = 1}^I \\prod_{j = 1}^{J_i} \\prod_{k = 1}^K\n", " \\boldsymbol{\\lambda}_{p_{ij} k}^{\\delta\\left[ f_{ij} - k \\right]}\n", " & \\quad & \\text{(3.9), (3.8)}\\\\\n", " &= \\left(\n", " \\prod_{m = 1}^M\n", " \\frac{\\Gamma\\left[K \\beta\\right]}{\\Gamma[\\beta]^K}\n", " \\prod_{k = 1}^K \\boldsymbol{\\lambda}_{mk}^{\\beta - 1}\n", " \\right)\n", " \\cdot\n", " \\prod_{m = 1}^M \\prod_{k = 1}^K \\boldsymbol{\\lambda}_{mk}^{F_{mk}}\n", " & \\quad & F_{mk} =\n", " \\sum_{i = 1}^I \\sum_{j = 1}^{J_i}\n", " \\delta\\left[ f_{ij} - k \\right]\n", " \\delta\\left[ p_{ij} - m \\right]\\\\\n", " &= \\prod_{m = 1}^M\n", " \\frac{\\Gamma\\left[K \\beta\\right]}{\\Gamma[\\beta]^K}\n", " \\prod_{k = 1}^K \\boldsymbol{\\lambda}_{mk}^{\\beta - 1 + F_{mk}}\\\\\n", " &= \\prod_{m = 1}^M\n", " \\frac{\\Gamma\\left[K \\beta\\right]}{\\Gamma[\\beta]^K}\n", " \\frac{\n", " \\prod_{k = 1}^K \\Gamma[\\beta + F_{mk}]\n", " }{\n", " \\Gamma\\left[ K \\beta + \\sum_{k = 1}^K F_{mk} \\right]\n", " }\n", " \\DirDist_{\\boldsymbol{\\lambda}_m}\\left[\n", " \\beta + F_{m1}, \\ldots, \\beta + F_{mK}\n", " \\right]\n", "\n", "(b)\n", "---\n", "\n", ".. math::\n", "\n", " \\prod_{i = 1}^I Pr(\\boldsymbol{\\pi}_i \\mid \\boldsymbol{\\alpha})\n", " \\prod_{j = 1}^{J_i} Pr(p_{ij} \\mid \\boldsymbol{\\pi}_i)\n", " &= \\prod_{i = 1}^I \\DirDist_{\\boldsymbol{\\pi}_i}[\\boldsymbol{\\alpha}]\n", " \\prod_{j = 1}^{J_i} \\CatDist_{p_{ij}}[\\boldsymbol{\\pi}_i]\n", " & \\quad & \\text{(20.5), (20.7)}\\\\\n", " &= \\prod_{i = 1}^I \\left(\n", " \\frac{\n", " \\Gamma\\left[ \\sum_{m = 1}^M \\boldsymbol{\\alpha}_m \\right]\n", " }{\n", " \\prod_{m = 1}^M \\Gamma\\left[ \\boldsymbol{\\alpha}_m \\right]\n", " }\n", " \\prod_{m = 1}^M \\boldsymbol{\\pi}_{im}^{\\boldsymbol{\\alpha}_m - 1}\n", " \\cdot\n", " \\prod_{j = 1}^{J_i} \\prod_{m = 1}^M\n", " \\boldsymbol{\\pi}_{im}^{\\delta\\left[ p_{ij} - m \\right]}\n", " \\right)\n", " & \\quad & \\text{(3.9), (3.8)}\\\\\n", " &= \\prod_{i = 1}^I \\left(\n", " \\frac{\\Gamma[M \\alpha]}{\\Gamma[\\alpha]^M}\n", " \\prod_{m = 1}^M \\boldsymbol{\\pi}_{im}^{\\alpha - 1}\n", " \\cdot\n", " \\prod_{m = 1}^M \\boldsymbol{\\pi}_{im}^{P_{im}}\n", " \\right)\n", " & \\quad & P_{im} = \\sum_{j = 1}^{J_i} \\delta[p_{ij} - m]\\\\\n", " &= \\prod_{i = 1}^I\n", " \\frac{\\Gamma\\left[M \\alpha \\right]}{\\Gamma[\\alpha]^M}\n", " \\prod_{m = 1}^M \\boldsymbol{\\pi}_{im}^{\\alpha - 1 + P_{im}}\\\\\n", " &= \\prod_{i = 1}^I\n", " \\frac{\\Gamma[M \\alpha]}{\\Gamma[\\alpha]^M}\n", " \\frac{\n", " \\prod_{m = 1}^M \\Gamma\\left[ \\alpha + P_{im} \\right]\n", " }{\n", " \\Gamma\\left[ M \\alpha + \\sum_{m = 1}^M P_{im} \\right]\n", " }\n", " \\DirDist_{\\boldsymbol{\\pi}_i}\\left[\n", " \\alpha + P_{i1}, \\ldots, \\alpha + P_{iM}\n", " \\right]" ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "Exercise 20.4\n", "=============\n", "\n", "Adapting LDA to the single author topic model in :cite:`fei2005bayesian`\n", "(Figure 3) results in the following generative process.\n", "\n", "Each image has only one object.\n", "For each image indexed by :math:`i \\in \\{1, \\ldots, I\\}` in a corpus:\n", "\n", "1. Choose an object :math:`w_i \\in \\{1, \\ldots, N\\}` from the distribution\n", " :math:`Pr(w_i) = \\CatDist_{w_i}[\\boldsymbol{\\nu}]`.\n", "#. Choose a :math:`M`-dimensional part probability vector\n", " :math:`\\boldsymbol{\\pi}_n` from the distribution\n", " :math:`Pr(\\boldsymbol{\\pi}_n \\mid w_i, \\boldsymbol{\\alpha}) =\n", " \\DirDist_{\\boldsymbol{\\pi}_n}[\\boldsymbol{\\alpha}_{w_i}]` where\n", " :math:`\\DeclareMathOperator{\\GammaDist}{Gamma}\n", " Pr(\\boldsymbol{\\alpha}_{w_i}) =\n", " \\GammaDist_{\\boldsymbol{\\alpha}_{w_i}}[\\kappa, \\theta]`.\n", "#. Choose :math:`J_i \\sim \\PoissonDist(\\xi)`.\n", "#. For each visual word indexed by :math:`j \\in \\{1, \\ldots, J_i\\}`\n", " in image :math:`i`:\n", "\n", " A. Choose a part :math:`p_{ij} \\in \\{1, \\ldots, M\\}` from the distribution\n", " :math:`Pr(p_{ij} \\mid \\boldsymbol{\\pi}_n) =\n", " \\CatDist_{p_{ij}}[\\boldsymbol{\\pi}_n]`.\n", " #. Choose a visual word :math:`f_{ij} \\in \\{1, \\ldots, K\\}` from the\n", " distribution\n", " :math:`Pr(f_{ij} \\mid p_{ij}, \\boldsymbol{\\lambda}) =\n", " \\CatDist_{f_{ij}}[\\boldsymbol{\\lambda}_{p_{ij}}]` where\n", " :math:`Pr(\\boldsymbol{\\lambda}_m) =\n", " \\DirDist_{\\boldsymbol{\\lambda}_m}[\\boldsymbol{\\beta}]`." ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ ".. _prince2012computer-ex-20.5:\n", "\n", "Exercise 20.5\n", "=============\n", "\n", "See Figure 1 of :cite:`rosen2004author`.\n", "\n", "The Author-Topic model observes the words and the set of authors per document;\n", "the other nodes are hidden. The generative process randomly selects an author\n", "and uses that author's (fixed) topic distribution as part of the word\n", "generation." ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "Exercise 20.6\n", "=============\n", "\n", "Notice that the parts in the constellation model (Figure 20.10) induces a\n", "spatial distribution over its associated words.\n", "\n", "Taking the graphical model of :ref:`Exercise 20.5 `\n", "as a template, the adjacent visual words can be encouraged to take the same part\n", "labels via adding a feature position indexed by the selected topic :math:`z`.\n", "In short, add another node with :math:`z` as its parent.\n", "\n", "The Gibbs sampling procedure will include an additional 2D normal distribution\n", "similar to (20.27)." ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "Exercise 20.7\n", "=============\n", "\n", "The graphical model can be constructed from the following generative process.\n", "\n", "For each image indexed by :math:`i \\in \\{1, \\ldots, I\\}` in a corpus:\n", "\n", "1. Choose a scene :math:`s_i \\in \\{1, \\ldots, C\\}` from the distribution\n", " :math:`Pr(s_i) = \\CatDist_{s_i}[\\boldsymbol{\\nu}]`.\n", "#. Choose :math:`J_i \\sim \\PoissonDist(\\xi)`.\n", "#. For each visual word indexed by :math:`j \\in \\{1, \\ldots, J_i\\}` in image\n", " :math:`i`:\n", "\n", " 1. Choose an object :math:`w_{ij} \\in \\{1, \\ldots, L\\}` from the distribution\n", " :math:`Pr(w_{ij} \\mid s_i = c) =\n", " \\CatDist_{w_{ij}}[\\boldsymbol{\\phi}_c]`.\n", " #. Choose a part :math:`p_{ij} \\in \\{1, \\ldots, M\\}` from the distribution\n", " :math:`Pr(p_{ij} \\mid w_{ij} = n) =\n", " \\CatDist_{p_{ij}}[\\boldsymbol{\\pi}_{w_n}]`.\n", " #. Choose a visual word :math:`f_{ij} \\in \\{1, \\ldots, K\\}` from the\n", " distribution\n", " :math:`Pr(f_{ij} \\mid p_{ij} = m) =\n", " \\CatDist_{f_{ij}}[\\boldsymbol{\\lambda}_m]`.\n", " #. Choose a feature position :math:`\\mathbf{x}_{ij}` from the distribution\n", " :math:`\\DeclareMathOperator{\\NormDist}{Norm}\n", " Pr(\\mathbf{x}_{ij} \\mid p_{ij} = m, w_{ij} = n) =\n", " \\NormDist_{\\mathbf{x}_{ij}}\\left[\n", " \\boldsymbol{\\mu}_n^{(w)} + \\boldsymbol{\\mu}_m^{(p)},\n", " \\boldsymbol{\\Sigma}_n^{(w)} + \\boldsymbol{\\Sigma}_m^{(p)}\n", " \\right]`.\n", "\n", ":math:`\\{ s_i \\}_{i = 1}^I, \\{ f_{ij} \\}_{i = 1, j = 1}^{I, J_i},\n", "\\{ x_{ij} \\}_{i = 1, j = 1}^{I, J_i}` are observable (manifest) variables.\n", "\n", ":math:`\\{ w_{ij} \\}_{i = 1, j = 1}^{I, J_i},\n", "\\{ p_{ij} \\}_{i = 1, j = 1}^{I, J_i}` are unobservable (latent) variables.\n", "\n", "The rest are parameters to be estimated with fixed hyperparameters." ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "Exercise 20.8\n", "=============\n", "\n", "Slides 2 to 9 of :cite:`lauritzen2007lvmfa` classifies latent variable models as\n", "\n", ".. table::\n", "\n", " +-----------------+-------------------------------------------------+\n", " | | Manifest variable |\n", " +-----------------+-------------------------+-----------------------+\n", " | Latent variable | Metrical | Categorical |\n", " +-----------------+-------------------------+-----------------------+\n", " | Metrical | Factor analysis | Latent trait analysis |\n", " +-----------------+-------------------------+-----------------------+\n", " | Categorical | Latent profile analysis | Latent class analysis |\n", " +-----------------+-------------------------+-----------------------+\n", "\n", "Discrete factor analysis is another name for latent trait analysis. Categorical\n", "variables can be either ordinal or nominal, and metrical variables can either be\n", "discrete or continuous.\n", "\n", "See :cite:`bartholomew1980factor` for more details." ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ ".. rubric:: References\n", "\n", ".. bibliography:: chapter-20.bib" ] } ], "metadata": { "anaconda-cloud": {}, "celltoolbar": "Raw Cell Format", "kernelspec": { "display_name": "Python [default]", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "nbformat_minor": 0 }