Nothing Special   »   [go: up one dir, main page]
Scribd Logo
Upload

Welcome to Scribd!

  • 20 views

    02 Infoth

    Uploaded by

    Tayyib
    1. Communication systems aim to convey messages from a transmitter to receiver but are limited by available power, noise, and bandwidth. The receiver's goal is to identify which message was sent, not necessarily what the exact message was. 2. The amount of information conveyed when a message is correctly identified is defined as the log of the inverse probability of that message. The average information across all messages is called the entropy, which is maximized when all messages are equally likely. 3. The information rate is the average number of bits of information that can be transmitted per second, calculated as the entropy multiplied by the message generation rate.

    Copyright:

    Attribution Non-Commercial (BY-NC)
    Download as PDF, TXT or read online from Scribd

    02 Infoth

    Uploaded by

    Tayyib
    20 views5 pages
    1. Communication systems aim to convey messages from a transmitter to receiver but are limited by available power, noise, and bandwidth. The receiver's goal is to identify which message was sent, not necessarily what the exact message was. 2. The amount of information conveyed when a message is correctly identified is defined as the log of the inverse probability of that message. The average information across all messages is called the entropy, which is maximized when all messages are equally likely. 3. The information rate is the average number of bits of information that can be transmitted per second, calculated as the entropy multiplied by the message generation rate.

    Original Description:

    Original Title

    02_infoth

    Copyright

    © Attribution Non-Commercial (BY-NC)

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    1. Communication systems aim to convey messages from a transmitter to receiver but are limited by available power, noise, and bandwidth. The receiver's goal is to identify which message was sent, not necessarily what the exact message was. 2. The amount of information conveyed when a message is correctly identified is defined as the log of the inverse probability of that message. The average information across all messages is called the entropy, which is maximized when all messages are equally likely. 3. The information rate is the average number of bits of information that can be transmitted per second, calculated as the entropy multiplied by the message generation rate.

    Copyright:

    Attribution Non-Commercial (BY-NC)
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    20 views5 pages

    02 Infoth

    Uploaded by

    Tayyib
    1. Communication systems aim to convey messages from a transmitter to receiver but are limited by available power, noise, and bandwidth. The receiver's goal is to identify which message was sent, not necessarily what the exact message was. 2. The amount of information conveyed when a message is correctly identified is defined as the log of the inverse probability of that message. The average information across all messages is called the entropy, which is maximized when all messages are equally likely. 3. The information rate is the average number of bits of information that can be transmitted per second, calculated as the entropy multiplied by the message generation rate.

    Copyright:

    Attribution Non-Commercial (BY-NC)
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    of 5

    Basic information theory

    Communication system performance is limited by Available signal power Background noise Bandwidth limits. Can we postulate an ideal system based on physical principles, against which we can assess actual systems? The role of a communication system is to convey, from transmitter to receiver, a sequence of messages selected from a nite number of possible messages. Within a specied time interval, one of these messages is transmitted, during the next another (maybe the same one), and so on. The messages are predetermined and known by the receiver The message selected for transmission during a particular interval is not known by the receiver The receiver knows the (different) probabilities for the selection of each message by the transmitter. The job of the receiver is not to answer the question What was the message?, but rather Which one?.

    1 Amount of information
    Suppose the allowable messages (or symbols) are m1, m2, . . . , and each have probability of occurrence p1 , p 2 , . . . .

    The transmitter selects message k with probability pk . (The complete set of symbols {m 1 , m 2 , . . .} is called the alphabet.) If the receiver correctly identies the message then an amount of information Ik given by 1 Ik log2 pk has been conveyed. Ik is dimensionless, but is measured in bits. Example: Given two equiprobable symbols m 1 and m 2 . Then p1 = 1/2, so correctly identifying m 1 carries I1 = log2 2 = 1bit of information. Similarly the correct transmission of m 2 carries 1 bit of information. Sometimes bit is used as an abbreviation for the term binary digit. If in the above example m 1 is represented by a 0 and m 2 by a 1, then we see that one binary digit carries 1 bit of information. This is not always the case use the term binit for binary digit when confusion can occur. Example: Suppose the binits 0 and 1 occur with probabilities 1/4 and 3/4 respectively. Then binit 0 carries log2 4 = 2 bits of information, and binit 1 carries log2 4/3 = 0.42 bits. The denition of information satises a number of useful criteria: It is intuitive: the occurrence of a highly probable event carries little information (Ik = 0 for pk = 1). It is positive: information may not decrease upon receiving a message (Ik 0 for 0 pk 1). We gain more information when a less probable message is received (Ik > Il for pk < pl ).

    Information is additive if the messages are independent: Ik,l = log2 1 1 1 = log2 + log2 = I k + Il . pk pl pk pl

    2 Average information: entropy


    Suppose we have M different independent messages (as before), and that a long sequence of L messages is generated. In the L message sequence, we expect p1 L occurrences of m 1 , p2 L of m 2 , etc. The total information in the sequence is Itotal = p1 L log2 1 1 + p2 L log2 + p1 p2

    so the average information per message interval will be Itotal 1 1 H= = p1 log2 + p2 log2 + = L p1 p2
    M

    pk log2
    k=1

    1 . pk

    This average information is referred to as the entropy. Example: Consider the case of just two messages with probabilities p and (1 p). This is called a binary memoryless source, since the successive symbols emitted are statistically independent. The average information per message is H = p log2 1 1 + (1 p) log2 p 1 p

    0.5

    0 0 0.2 0.4 p 0.6 0.8 1

    The average information is maximised for p = 1/2, with a corresponding entropy of 1 bit/message. In general, it can be proved that for M messages, the entropy H becomes maximum when all messages are equally likely. Then each message has probability 1/M, and the entropy is
    M

    Hmax =
    k=1

    1 log2 M = log2 M. M

    3 Information rate
    If the source of the messages generates messages at the rate r per second, then the information rate is dened to be R r H = average number of bits of information per second. Example: An analogue signal is band limited to B Hz, sampled at the Nyquist rate, and the samples quantised to 4 levels. The quantisation levels Q 1 , Q 2 , Q 3 , and Q 4 (messages) are assumed independent, and occur with probabilities p1 = p4 = 1/8 and p2 = p3 = 3/8.

    The average information per symbol at the source is


    4

    H=
    k=1

    pk log2

    1 pk

    3 8 3 8 1 1 log2 8 + log2 + log2 + log2 8 8 8 3 8 3 8 = 1.8 bits/message. = The information rate R is R = r H = 2B(1.8) = 3.6B bits/s. Note that we could identify each message by a 2 digit binary code: Message Q1 Q2 Q3 Q4 Probability 1/8 3/8 3/8 1/8 Binary code 0 0 0 1 1 0 1 1

    If we transmit 2B messages per second, we will be transmitting 4B binits per second (since each message requires 2 binits). Since each binit should be able to carry 1 bit of information, we should be able to transmit 4B bits of information using 4B binits. From the example we see that we are only transmitting 3.6B bits. We are therefore not taking full advantage of the ability of binary PCM to convey information. One remedy is to select different quantisation levels, such that each level is equally likely. Other methods involve the use of more complicated code assignments.

    You might also like