need to see the descriptions of logging output of various modules in python -

i want create log file in addition standard output. using inbuilt module such as:

def dedup_col_col2(final_recos):     """     removing duplicate combination of col1-col2 single muid     """     recommendation_log()     def f(data,col1, col2):         ra=data[~(data[[col1,col2]].duplicated())]         return ra     .....do something(carry on)  def function_logger(file_level, console_level = none):     """     logging function log warning/error messages     writes both console , log file     """     function_name = inspect.stack()[1][3]     logger = logging.getlogger(function_name)     logger.setlevel(logging.debug) #by default, logs messages     if console_level != none:         ch = logging.streamhandler() #streamhandler logs console         ch.setlevel(console_level)         ch_format = logging.formatter('%(asctime)s - %(message)s')         ch.setformatter(ch_format)         logger.addhandler(ch)     fh = logging.filehandler("{0}.log".format(function_name))     fh.setlevel(file_level)     fh_format = logging.formatter('%(asctime)s - %(lineno)d - %(levelname)-8s - %(message)s')     fh.setformatter(fh_format)     logger.addhandler(fh)     return logger  def recommendation_log():     recommendation_log_logger = function_logger(logging.debug, logging.debug)     recommendation_log_logger.debug('debug message')     recommendation_log_logger.info('info message')     recommendation_log_logger.warn('warn message')     recommendation_log_logger.error('error message')     recommendation_log_logger.critical('critical message')     def main():     recommendation_log()       final_recos1=dedup_tbid_tbidrank(final_recos)     logging.shutdown() 

code taken following link: how write log messages log file , console @ same time?

now when execute code shows various debug-error messages on console. when open log file, shows me following output in recommendation_log.log file:

2015-07-07 12:19:16,392 - 339 - debug    - debug message 2015-07-07 12:19:16,392 - 340 - info     - info message 2015-07-07 12:19:16,393 - 341 - warning  - warn message 2015-07-07 12:19:16,393 - 342 - error    - error message 2015-07-07 12:19:16,393 - 343 - critical - critical message 2015-07-07 12:22:03,176 - 339 - debug    - debug message 2015-07-07 12:22:03,176 - 339 - debug    - debug message 

now want understand source of these errors , want see descriptions of these messages, in same way shown on console.

edit: on console displays me warnings such as:

***cmeans_omni.py:37: settingwithcopywarning:  value trying set on copy of slice dataframe. try using .loc[row_indexer,col_indexer] = value instead see the caveats in documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy   df['campaignname_upd'] = df['campaignname'] /home/ubuntu/anaconda/lib/python2.7/site-packages/pandas/io/parsers.py:1170: dtypewarning: columns (3,7) have mixed types. specify dtype option on import or set low_memory=false.   data = self._reader.read(nrows)*** 

now want understand source of these errors , want see descriptions of these messages in log file, in same way shown on console.

basically want see description, stated above in log file , actual line no , originating. line number, shown in sample output not actual line number, line number of line corresponding error in function.

i have been reading logging module day long, cannot figure out solution.

you mixing 2 things.

let's first concentrate on logging. approach seems work, looks quite messy. same idea, less overhead:

import logging  logger = logging.getlogger(__name__) logger.setlevel(logging.debug)  fmt = logging.formatter("%(asctime)s - "                         "%(module)s:%(lineno)s - "                         "%(levelname)s: %(message)s")  # loggers need handlers handling log messages. # lets's add streamhandler console output... sh = logging.streamhandler() sh.setformatter(fmt) sh.setlevel(logging.debug) logger.addhandler(sh)  # ...and filehandler logfile writing: fh = logging.filehandler('log.file', 'a') fh.setformatter(fmt) fh.setlevel(logging.warn) logger.addhandler(fh)  # let's log stuff: logger.debug("foobar") logger.info("meh") logger.warn("i warned you.") logger.error("this might error.") 

if encounter errors , want include traceback in log, use logger.exception():

try:     x = 1 / 0 except zerodivisionerror e:     # includes traceback level error     logger.exception(e) 

however, error warning, not force program terminate. if need warning-related stuff, might want take @ warnings module.

in case, pandas throws settingwithcopywarning, assigning values copy of part of dataframe. means, modify not affect original data, since working copy only. use df.loc[] as explained in documentation past this.

---

update:

using warnings.warn() messages, add them logger using catch_warnings()-contextmanager:

import warnings  class warningsadapter(logging.loggeradapter):     """copy `warnings.warningmessage` information log record"""     def process(self, msg, kwargs):         if isinstance(msg, warnings.warningmessage):             msg = "%s: %s" % (msg.category.__name__, msg.message)         return msg, kwargs  adapter = warningsadapter(logger, extra=none)  warnings.catch_warnings(record=true) warning_list:     # throwing warning     warnings.warn("this warning message.", runtimewarning)     warnings.warn("something might have gone wrong.", userwarning)     # else     w in warning_list:         adapter.warning(w) 

you might want take @ this question though, use cases of warnings.warn() , logging.warning() being discussed there.